VrlWiki - User contributions [en]

CS295J/Final contributions

2009-05-01T16:53:22Z

Eric Sodomka: /* Parallel Framework for Evaluation Modules */

= Specific Contributions =

[I have removed almost everything that does not describe itself as a contribution. If your part is missing, a revision is in order and you can find what the former version on the page linked below (David)]

[[CS295J/Thursday 2pm version of final contribution|Thursday 2pm version of final contribution]]

== Workflow, Multi-tasking and Interruptions ==
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Aims'''
* Develop a Theory of Multi-tasking Workflow and the Impact of Interruptions
* Build a Meta-work Assistance Tool
:# We will perform a series of ecologically-valid studies to compare user performance between a state of the art/popular task management system (control group) and our meta-work assistance tool (experimental group) [Didn't your study show that there is nothing close to a killer app in this department? - Steven] [By "state of the art" I mean a popular, well-recognized system (or systems), such as Outlook, that are used by large numbers of people.]
* Evaluate the Module-based Model of HCI by using it to predict the outcome of the above-mentioned evaluation; then compare with empirical results

'''Contributions'''
* Theory of Multi-tasking Workflow and the Impact of Interruptions
:# We will undertake detailed studies to help understand the following questions:
::# How does the size of a user's working set impact interruption resumption time?
::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:* Internal dependencies: none
* Empirical Evaluation of Meta-work Assistance Tool in an Ecologically-valid Context
:* Internal dependencies: Completion of a testable meta-work tool
* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:* Internal dependencies: Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

'''Demonstration (we might consider calling these deliverables)'''
* A predictive, quantiative model of controlled task performance and interruption resumption time for varying sized working sets
* An ecologically valid evaluation which shows that the completed meta-work assistance tool outperforms a state of the art system such as Outlook
* A comparison between the result predicted by the module-based model of HCI and the evaluation mentioned above; accompanied by a detailed analysis of factors contributing to the result

'''External Dependencies'''
* Dependent on most of the rest of the proposal being completed to the point of being usable to predict the outcome of the meta-work tool evaluation

'''Background and Related Work'''
* Salvucci et al's integrated theory of multi-tasking continuum integrates concurrent and sequential task performance
* Gloria Mark's research group has conducted a number of empirical studies into multi-tasking and interruption

'''Preliminary Work'''

* [[Study of Meta-work Tools and Strategies of 7 Information Workers]]
:* Results suggest there is not a single "ideal" meta-work tool today, motivating our proposal to build a new meta-work support tool based in part on a scientific theory of multi-tasking and interruptions.

'''Reviews'''
* Not in the expected format -- split into appropriate parts and label them as in the other sections (David)

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.[How do you propose to get around the problem Andrew brought up of accurately linking this low-level data to the interface/program's state at the moment it occurred? - Steven] [Low-level interactions will need to be logged along with a markup-language-style state description of the application itself. This would either require access to the source code, or a detailed interface description from the interface designer that would allow for complete simulation of the application.]
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.[Perhaps something should be said about sectioning this data off into modules - i.e. compiling all e-mail writing recordings to analyze separately from e-mail sorting recordings in order to facilitate analysis. - Steven] [I believe this is addressed in the "semantic-level chunking" section, given below.]

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)] [Things like pupil-tracking can be quantitatively evaluated through users tests. For instance, we could instruct users to shift the focus of their eyes from one on-screen object to another as quickly as possible some fixed number of times. We could then compare the results from our system to determine accuracy and speed, and by varying the size and distance of the target objects, we could obtain richer data for analysis. With respect to clustering low-level interactions, I believe that issue is addressed in the following section.][Should this take into account the need to match recordings to interface state? - Steven] [Possibly, if we go down the simulation route. If we're actually logging all of the events from within the source code itself, I believe the demonstration of syncing low-level interaction with application-state is self-evident.]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>. Can you briefly summarize the utility with one sentence? Or is that what the next bullet does? (Jon) [In the instance of this work, utility was demonstrated with respect to automatically generating presentations from data explorations, and providing quantitative interface-usage analysis that aided future decisions for interface designers. i.e., this operations was performed a lot, and this one was not; therefore we should focus on making the first more efficient, perhaps at the expense of the second. The results of the work are fairly basic, but they do demonstrate that logging histories is a useful practice for a number of reasons.]
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)
* Two thumbs up. - Steven

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)] [I don't intend for these techniques to be performed manually. Application-specific rules for chunking will be created manually, most likely by the interface designer, but those rules will be applied automatically to filter low-level primitives and application-states into semantic-level interactions. The success of such methods could be evaluated by having automatically-generated semantic-level histories compared to those manually created by a user. Of course, there's no guarantee that what a user manually creates is any good, so I think it would be more effective to have a quantitative study of user performance in which timed-tasks are completed with and without the ability to review and revisit semantic-level histories. The test without any history at all would serve as a baseline, and then performance with histories created using different chunking rules would speak to the effectiveness of different strategies for chunking.]

'''Dependencies'''
* Software framework for collecting user interactions.[I'd like to see this fleshed out in a theoretical manner. - Steven] [I believe the previous section speaks to this question, but perhaps not with enough detail. It seems appropriate to have this discussion in class.]
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon) [This is a great question. It really depends on what's meant by accuracy. If the only level of semantics we want is that defined by the interface-designer, then it's not really an interesting question. As we begin to talk about semantics with respect to what the user ''perceives'' or what the user's ''intent'' might be, it becomes more interesting, and the term accuracy becomes more ambiguous. I suppose accuracy could be tested by having users verbally speak their intent as they use an interface, and compare their descriptions with the output of the system, but I think that might be a messy study. Interesting to think about.]
Can you include something about the broader impact of this contribution? (Jon) [I think the broader impact here is offering interface designers tools to analyze interaction at a high enough level that they can begin to observe user's intent with an interface, and potentially infer overarching user-goals. If these tools prove effective, and could be implemented in real-time, there's a wealth of possibility in adaptive interfaces. In addition, the semantic-level interaction history construct has the potential to provide meaningful training data for machine-learning algorithms that could probabilistically model user-action and be used to offer suggested interactions.]

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A framework for the unification of established heuristic usability guidelines and accepted cognitive principles. We propose a matrix pairing cognitive principles and heuristic guidelines into specific interface improvement recommendations.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

[Language cleaned up. While I still think it is important to give more "weight" to cognitive/heuristic principle pairs that agree more strongly than others and output more useful recommendations, I believe I've pitched it wrong and made it too prominent an attribute of this module. I've adjusted. [[User:E J Kalafarski|E J Kalafarski]] 15:34, 1 May 2009 (UTC)]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A human "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A human "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several recommendations from a matrix of cogntive/heuristic principle pairs.
* User testing demonstrates the assumed efficacy and applicability of the principle pairs versus separate application of cognitive and heuristic guidelines.
* Matrix can be improved incrementally by applying "weight" to each cell of the matrix, increasing its influence on final output recommendations, based on the measurable success of this study.

[There is still the question of how these weights are determined. Is this the job of the third expert? Or is the third expert given these weights, and he just determines how to apply them? -Eric] [Added more explanatory description of methodology as a bullet under Demo. [[User:E J Kalafarski|E J Kalafarski]] 15:49, 1 May 2009 (UTC)]

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon) [Yes. [[User:E J Kalafarski|E J Kalafarski]] 15:49, 1 May 2009 (UTC)]

* Gideon's comment brings up an important point - in what form will the weights be presented? - Steven [Values which contribute to the "influence" of each recommendation. [[User:E J Kalafarski|E J Kalafarski]] 15:49, 1 May 2009 (UTC)]
* Could be more explicit about whether the experts are simulated agents or real people (Jon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a set of interface evaluation modules for analysis, we demonstrate the proposed efficiency and accuracy of a method for aggregating these interface suggestions. In this preliminary demonstration, we limit the field to a strictly-defined interface and 2–3 CPM principles (e.g. Fitts' Law and Affordance). Unified recommendations are generated for application to the interface for incremental improvement in usability.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A limited but strictly-controlled study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

'''Dependency''': Framework for the input of a given interface to individual modules, and standardization of the output of said modules.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon) [I limited the scope of this initial demonstration. [[User:E J Kalafarski|E J Kalafarski]] 15:46, 1 May 2009 (UTC)]
* [I think this is the same as my contribution, and that we should merge them together. You've covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation. You also gave a demonstration of comparing recommendations, while I only covered evaluations. -Eric]
* I think this contribution and mine (the CPM-GOMS improvement) will need to be reconciled into a single unit. - Steven
[Is it feasible to combine all three of these contributions? [[User:E J Kalafarski|E J Kalafarski]] 15:47, 1 May 2009 (UTC)]
* Interesting to note that this could describe the overarching architecture but it depends upon components of the architecture to work. Would it depend on any other modules? (Jon) [Yes, several modules are listed as dependencies. [[User:E J Kalafarski|E J Kalafarski]] 15:46, 1 May 2009 (UTC)]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# In user studies, this module predicted time-to-completion more accurately than the standard CPM-GOMS model.
:::* Dependencies
:::# CPM-GOMS Module with proposed cognitive load extension
::* Cognitive Load
:::# User studies were conducted during which participants brain's were scanned with EEG to measure activation levels.
:::# Self-report measures were obtained describing the amount of concentration/effort required on the part of the user.
:::# The literature base provided an accurate assessment of cognitive load in the tasks involved.
:::# Facial recognition allows us to approximate spans of high-load (e.g., furled brow)
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::::# Self-Report Measure
::::# Database of load per cognitive tasks
::* Frustration
:::# Frustration is measured by asking participants to respond on the self-report scale.
:::# Facial recognition allows us to approximate spans of frustration (e.g., frowning, sighing, furled brow)
:::# Galvanic Skin Response allows detection of emotional arousal
:::# Heart rate monitoring allows detection of emotional arousal
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response input
::::# Heart rate response input
::::# Interface Efficiency Module

'''review'''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)] [I agree that this pretty close to E.J.'s contribution, and that maybe they should be merged. E.J. covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation, and his gave a demonstration of comparing recommendations, while I only covered evaluations.]
* I feel like this structure's clarity may come at expense of specificity, I'd like to know more than that it's parallel. - Steven
* Very thorough demonstration(s)! (Jon)
* I think that the contribution is much more clearly stated here -- I would argue for integrating them together. (Andrew)

== Anti-Pattern Recognition & Resolution ==
Owner: [[User:Ian Spector | Ian Spector]]

''Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation. An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.''

'''Aims:'''
* Develop a framework for analyzing an interface and recognizing violations of established patterns of possible interaction
* Develop an algorithm to quantitatively rate the amount of anti-pattern violations
* Integrate cognitive theory in providing suitable alternatives

'''Contributions:'''
* Creation of established patterns of interaction as determined by user studies
* Creation of algorithm to detect match and library elements to interface elements

'''Demonstration:'''
* The viability of this module will be tested through providing it with a poorly designed interface.
** The design would be analyzed and tested by: a usability expert, normal users, the module whereby each would have to eventually provide a value as to how much the interface exhibits anti-patterns.
** Based on the results provided, we should expect to see the expert and module reports to be similar

'''Dependencies:'''
* Library of standard design patterns
* Affordances module
* Custom design pattern libraries (optional)

'''Inputs:'''
* Formal interface description
* Tasks which can be performed within the interface

'''Outputs:'''
* Identification of interface elements whose placement or function are contrary to the pattern library
* Recommendations for alternative functionality or placement of such elements.

== Sample Modules ==

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs [In theory could the non-semantic inputs be automatically read from the interface? - Steven]
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task [Could this module benefit from eye-tracking? - Steven]

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

CS295J/Final contributions

2009-05-01T16:53:06Z

Eric Sodomka: /* Parallel Framework for Evaluation Modules */

= Specific Contributions =

[I have removed almost everything that does not describe itself as a contribution. If your part is missing, a revision is in order and you can find what the former version on the page linked below (David)]

[[CS295J/Thursday 2pm version of final contribution|Thursday 2pm version of final contribution]]

== Workflow, Multi-tasking and Interruptions ==
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Aims'''
* Develop a Theory of Multi-tasking Workflow and the Impact of Interruptions
* Build a Meta-work Assistance Tool
:# We will perform a series of ecologically-valid studies to compare user performance between a state of the art/popular task management system (control group) and our meta-work assistance tool (experimental group) [Didn't your study show that there is nothing close to a killer app in this department? - Steven] [By "state of the art" I mean a popular, well-recognized system (or systems), such as Outlook, that are used by large numbers of people.]
* Evaluate the Module-based Model of HCI by using it to predict the outcome of the above-mentioned evaluation; then compare with empirical results

'''Contributions'''
* Theory of Multi-tasking Workflow and the Impact of Interruptions
:# We will undertake detailed studies to help understand the following questions:
::# How does the size of a user's working set impact interruption resumption time?
::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:* Internal dependencies: none
* Empirical Evaluation of Meta-work Assistance Tool in an Ecologically-valid Context
:* Internal dependencies: Completion of a testable meta-work tool
* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:* Internal dependencies: Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

'''Demonstration (we might consider calling these deliverables)'''
* A predictive, quantiative model of controlled task performance and interruption resumption time for varying sized working sets
* An ecologically valid evaluation which shows that the completed meta-work assistance tool outperforms a state of the art system such as Outlook
* A comparison between the result predicted by the module-based model of HCI and the evaluation mentioned above; accompanied by a detailed analysis of factors contributing to the result

'''External Dependencies'''
* Dependent on most of the rest of the proposal being completed to the point of being usable to predict the outcome of the meta-work tool evaluation

'''Background and Related Work'''
* Salvucci et al's integrated theory of multi-tasking continuum integrates concurrent and sequential task performance
* Gloria Mark's research group has conducted a number of empirical studies into multi-tasking and interruption

'''Preliminary Work'''

* [[Study of Meta-work Tools and Strategies of 7 Information Workers]]
:* Results suggest there is not a single "ideal" meta-work tool today, motivating our proposal to build a new meta-work support tool based in part on a scientific theory of multi-tasking and interruptions.

'''Reviews'''
* Not in the expected format -- split into appropriate parts and label them as in the other sections (David)

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.[How do you propose to get around the problem Andrew brought up of accurately linking this low-level data to the interface/program's state at the moment it occurred? - Steven] [Low-level interactions will need to be logged along with a markup-language-style state description of the application itself. This would either require access to the source code, or a detailed interface description from the interface designer that would allow for complete simulation of the application.]
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.[Perhaps something should be said about sectioning this data off into modules - i.e. compiling all e-mail writing recordings to analyze separately from e-mail sorting recordings in order to facilitate analysis. - Steven] [I believe this is addressed in the "semantic-level chunking" section, given below.]

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)] [Things like pupil-tracking can be quantitatively evaluated through users tests. For instance, we could instruct users to shift the focus of their eyes from one on-screen object to another as quickly as possible some fixed number of times. We could then compare the results from our system to determine accuracy and speed, and by varying the size and distance of the target objects, we could obtain richer data for analysis. With respect to clustering low-level interactions, I believe that issue is addressed in the following section.][Should this take into account the need to match recordings to interface state? - Steven] [Possibly, if we go down the simulation route. If we're actually logging all of the events from within the source code itself, I believe the demonstration of syncing low-level interaction with application-state is self-evident.]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>. Can you briefly summarize the utility with one sentence? Or is that what the next bullet does? (Jon) [In the instance of this work, utility was demonstrated with respect to automatically generating presentations from data explorations, and providing quantitative interface-usage analysis that aided future decisions for interface designers. i.e., this operations was performed a lot, and this one was not; therefore we should focus on making the first more efficient, perhaps at the expense of the second. The results of the work are fairly basic, but they do demonstrate that logging histories is a useful practice for a number of reasons.]
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)
* Two thumbs up. - Steven

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)] [I don't intend for these techniques to be performed manually. Application-specific rules for chunking will be created manually, most likely by the interface designer, but those rules will be applied automatically to filter low-level primitives and application-states into semantic-level interactions. The success of such methods could be evaluated by having automatically-generated semantic-level histories compared to those manually created by a user. Of course, there's no guarantee that what a user manually creates is any good, so I think it would be more effective to have a quantitative study of user performance in which timed-tasks are completed with and without the ability to review and revisit semantic-level histories. The test without any history at all would serve as a baseline, and then performance with histories created using different chunking rules would speak to the effectiveness of different strategies for chunking.]

'''Dependencies'''
* Software framework for collecting user interactions.[I'd like to see this fleshed out in a theoretical manner. - Steven] [I believe the previous section speaks to this question, but perhaps not with enough detail. It seems appropriate to have this discussion in class.]
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon) [This is a great question. It really depends on what's meant by accuracy. If the only level of semantics we want is that defined by the interface-designer, then it's not really an interesting question. As we begin to talk about semantics with respect to what the user ''perceives'' or what the user's ''intent'' might be, it becomes more interesting, and the term accuracy becomes more ambiguous. I suppose accuracy could be tested by having users verbally speak their intent as they use an interface, and compare their descriptions with the output of the system, but I think that might be a messy study. Interesting to think about.]
Can you include something about the broader impact of this contribution? (Jon) [I think the broader impact here is offering interface designers tools to analyze interaction at a high enough level that they can begin to observe user's intent with an interface, and potentially infer overarching user-goals. If these tools prove effective, and could be implemented in real-time, there's a wealth of possibility in adaptive interfaces. In addition, the semantic-level interaction history construct has the potential to provide meaningful training data for machine-learning algorithms that could probabilistically model user-action and be used to offer suggested interactions.]

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A framework for the unification of established heuristic usability guidelines and accepted cognitive principles. We propose a matrix pairing cognitive principles and heuristic guidelines into specific interface improvement recommendations.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

[Language cleaned up. While I still think it is important to give more "weight" to cognitive/heuristic principle pairs that agree more strongly than others and output more useful recommendations, I believe I've pitched it wrong and made it too prominent an attribute of this module. I've adjusted. [[User:E J Kalafarski|E J Kalafarski]] 15:34, 1 May 2009 (UTC)]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A human "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A human "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several recommendations from a matrix of cogntive/heuristic principle pairs.
* User testing demonstrates the assumed efficacy and applicability of the principle pairs versus separate application of cognitive and heuristic guidelines.
* Matrix can be improved incrementally by applying "weight" to each cell of the matrix, increasing its influence on final output recommendations, based on the measurable success of this study.

[There is still the question of how these weights are determined. Is this the job of the third expert? Or is the third expert given these weights, and he just determines how to apply them? -Eric] [Added more explanatory description of methodology as a bullet under Demo. [[User:E J Kalafarski|E J Kalafarski]] 15:49, 1 May 2009 (UTC)]

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon) [Yes. [[User:E J Kalafarski|E J Kalafarski]] 15:49, 1 May 2009 (UTC)]

* Gideon's comment brings up an important point - in what form will the weights be presented? - Steven [Values which contribute to the "influence" of each recommendation. [[User:E J Kalafarski|E J Kalafarski]] 15:49, 1 May 2009 (UTC)]
* Could be more explicit about whether the experts are simulated agents or real people (Jon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a set of interface evaluation modules for analysis, we demonstrate the proposed efficiency and accuracy of a method for aggregating these interface suggestions. In this preliminary demonstration, we limit the field to a strictly-defined interface and 2–3 CPM principles (e.g. Fitts' Law and Affordance). Unified recommendations are generated for application to the interface for incremental improvement in usability.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A limited but strictly-controlled study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

'''Dependency''': Framework for the input of a given interface to individual modules, and standardization of the output of said modules.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon) [I limited the scope of this initial demonstration. [[User:E J Kalafarski|E J Kalafarski]] 15:46, 1 May 2009 (UTC)]
* [I think this is the same as my contribution, and that we should merge them together. You've covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation. You also gave a demonstration of comparing recommendations, while I only covered evaluations. -Eric]
* I think this contribution and mine (the CPM-GOMS improvement) will need to be reconciled into a single unit. - Steven
[Is it feasible to combine all three of these contributions? [[User:E J Kalafarski|E J Kalafarski]] 15:47, 1 May 2009 (UTC)]
* Interesting to note that this could describe the overarching architecture but it depends upon components of the architecture to work. Would it depend on any other modules? (Jon) [Yes, several modules are listed as dependencies. [[User:E J Kalafarski|E J Kalafarski]] 15:46, 1 May 2009 (UTC)]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# In user studies, this module predicted time-to-completion more accurately than the standard CPM-GOMS model.
:::* Dependencies
:::# CPM-GOMS Module with proposed cognitive load extension
::* Cognitive Load
:::# User studies were conducted during which participants brain's were scanned with EEG to measure activation levels.
:::# Self-report measures were obtained describing the amount of concentration/effort required on the part of the user.
:::# The literature base provided an accurate assessment of cognitive load in the tasks involved.
:::# Facial recognition allows us to approximate spans of high-load (e.g., furled brow)
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::::# Self-Report Measure
::::# Database of load per cognitive tasks
::* Frustration
:::# Frustration is measured by asking participants to respond on the self-report scale.
:::# Facial recognition allows us to approximate spans of frustration (e.g., frowning, sighing, furled brow)
:::# Galvanic Skin Response allows detection of emotional arousal
:::# Heart rate monitoring allows detection of emotional arousal
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response input
::::# Heart rate response input
::::# Interface Efficiency Module

'''review'''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)] [I think this pretty close to E.J.'s contribution, and that maybe they should be merged. E.J. covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation, and his gave a demonstration of comparing recommendations, while I only covered evaluations.]
* I feel like this structure's clarity may come at expense of specificity, I'd like to know more than that it's parallel. - Steven
* Very thorough demonstration(s)! (Jon)
* I think that the contribution is much more clearly stated here -- I would argue for integrating them together. (Andrew)

== Anti-Pattern Recognition & Resolution ==
Owner: [[User:Ian Spector | Ian Spector]]

''Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation. An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.''

'''Aims:'''
* Develop a framework for analyzing an interface and recognizing violations of established patterns of possible interaction
* Develop an algorithm to quantitatively rate the amount of anti-pattern violations
* Integrate cognitive theory in providing suitable alternatives

'''Contributions:'''
* Creation of established patterns of interaction as determined by user studies
* Creation of algorithm to detect match and library elements to interface elements

'''Demonstration:'''
* The viability of this module will be tested through providing it with a poorly designed interface.
** The design would be analyzed and tested by: a usability expert, normal users, the module whereby each would have to eventually provide a value as to how much the interface exhibits anti-patterns.
** Based on the results provided, we should expect to see the expert and module reports to be similar

'''Dependencies:'''
* Library of standard design patterns
* Affordances module
* Custom design pattern libraries (optional)

'''Inputs:'''
* Formal interface description
* Tasks which can be performed within the interface

'''Outputs:'''
* Identification of interface elements whose placement or function are contrary to the pattern library
* Recommendations for alternative functionality or placement of such elements.

== Sample Modules ==

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs [In theory could the non-semantic inputs be automatically read from the interface? - Steven]
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task [Could this module benefit from eye-tracking? - Steven]

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

CS295J/Final contributions

2009-04-29T17:06:21Z

Eric Sodomka: /* Logic */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)]

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon)

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and [...HCI design principles? -Eric]
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

[There is still the question of how these weights are determined. Is this the job of the third expert? Or is the third expert given these weights, and he just determines how to apply them? -Eric]

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon)
* [I think this is the same as my contribution, and that we should merge them together. You've covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation. You also gave a demonstration of comparing recommendations, while I only covered evaluations. -Eric]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

[I agree that a metric (or several) is crucial to our framework, but what is your demonstration of the feasibility or usability of these particular values you've chosen. On an unrelated note, we talked a lot about how some of these might be "convertible" into others…do you see these metrics as different sides of the same coin? Can the units of one be "converted" into the units of another, like gallons to liters? [[User:E J Kalafarski|E J Kalafarski]] 15:10, 29 April 2009 (UTC)]

[More detail on each demonstration would be helpful. -Eric]

'''review'''

* Words cannot express how amazing this contribution is. It should be given as much money as requested without question.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

'''review'''

* Cool- I like the xml. (Gideon)
* [I really like the xml, too. I'm guessing you just have placeholder text for some of these values (e.g. "pulling hair out"), but maybe we should think more about what exactly is returned. I was thinking the evaluation values would be some sort of utility value between 0-1, but this makes the "time" dimension tricky (what does a "time" score of 0.75 mean?). -Eric] 

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

Evaluation method:
* Evaluate an interface via the proposed framework, as well as traditional CPM-GOMS and ACT-R methods. Also have a small team of interface design experts evaluate the interface. Solicit comments on the current interface and suggestions for improvement from each team/method, and compare the accuracy and validity of the results.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-29T17:06:08Z

Eric Sodomka: /* Logic */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)]

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon)

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and [...HCI design principles? -Eric]
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

[There is still the question of how these weights are determined. Is this the job of the third expert? Or is the third expert given these weights, and he just determines how to apply them? -Eric]

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon)
* [I think this is the same as my contribution, and that we should merge them together. You've covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation. You also gave a demonstration of comparing recommendations, while I only covered evaluations. -Eric]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

[I agree that a metric (or several) is crucial to our framework, but what is your demonstration of the feasibility or usability of these particular values you've chosen. On an unrelated note, we talked a lot about how some of these might be "convertible" into others…do you see these metrics as different sides of the same coin? Can the units of one be "converted" into the units of another, like gallons to liters? [[User:E J Kalafarski|E J Kalafarski]] 15:10, 29 April 2009 (UTC)]

[More detail on each demonstration would be helpful. -Eric]

'''review'''

* Words cannot express how amazing this contribution is. It should be given as much money as requested without question.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

'''review'''

* Cool- I like the xml. (Gideon)
* [I really like the xml, too. I'm guessing you just have placeholder text for some of these values (e.g. "pulling hair out"), but maybe we should think more about what exactly is returned. I was thinking the evaluation values would be some sort of utility value between 0-1, but this makes the "time" dimension tricky (what does a "time" score of 0.75 mean?).] 

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

Evaluation method:
* Evaluate an interface via the proposed framework, as well as traditional CPM-GOMS and ACT-R methods. Also have a small team of interface design experts evaluate the interface. Solicit comments on the current interface and suggestions for improvement from each team/method, and compare the accuracy and validity of the results.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-29T16:58:18Z

Eric Sodomka: /* Evaluation Metrics */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)]

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon)

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and [...HCI design principles? -Eric]
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

[There is still the question of how these weights are determined. Is this the job of the third expert? Or is the third expert given these weights, and he just determines how to apply them? -Eric]

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon)
* [I think this is the same as my contribution, and that we should merge them together. You've covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation. You also gave a demonstration of comparing recommendations, while I only covered evaluations. -Eric]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

[I agree that a metric (or several) is crucial to our framework, but what is your demonstration of the feasibility or usability of these particular values you've chosen. On an unrelated note, we talked a lot about how some of these might be "convertible" into others…do you see these metrics as different sides of the same coin? Can the units of one be "converted" into the units of another, like gallons to liters? [[User:E J Kalafarski|E J Kalafarski]] 15:10, 29 April 2009 (UTC)]

[More detail on each demonstration would be helpful. -Eric]

'''review'''

* Words cannot express how amazing this contribution is. It should be given as much money as requested without question.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

'''review'''

* Cool- I like the xml. (Gideon)

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

Evaluation method:
* Evaluate an interface via the proposed framework, as well as traditional CPM-GOMS and ACT-R methods. Also have a small team of interface design experts evaluate the interface. Solicit comments on the current interface and suggestions for improvement from each team/method, and compare the accuracy and validity of the results.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-29T16:52:48Z

Eric Sodomka: /* Evaluation for Recommendation and Incremental Improvement */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)]

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon)

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and [...HCI design principles? -Eric]
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

[There is still the question of how these weights are determined. Is this the job of the third expert? Or is the third expert given these weights, and he just determines how to apply them? -Eric]

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon)
* [I think this is the same as my contribution, and that we should merge them together. You've covered some things that I didn't, such as a more detailed demonstration of how our framework is better than a human interface designer's evaluation. You also gave a demonstration of comparing recommendations, while I only covered evaluations. -Eric]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

[I agree that a metric (or several) is crucial to our framework, but what is your demonstration of the feasibility or usability of these particular values you've chosen. On an unrelated note, we talked a lot about how some of these might be "convertible" into others…do you see these metrics as different sides of the same coin? Can the units of one be "converted" into the units of another, like gallons to liters? [[User:E J Kalafarski|E J Kalafarski]] 15:10, 29 April 2009 (UTC)]

'''review'''

* Words cannot express how amazing this contribution is. It should be given as much money as requested without question.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

'''review'''

* Cool- I like the xml. (Gideon)

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

Evaluation method:
* Evaluate an interface via the proposed framework, as well as traditional CPM-GOMS and ACT-R methods. Also have a small team of interface design experts evaluate the interface. Solicit comments on the current interface and suggestions for improvement from each team/method, and compare the accuracy and validity of the results.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-29T16:47:42Z

Eric Sodomka: /* Reconciling Usability Heuristics with Cognitive Theory */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)]

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon)

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and [...HCI design principles? -Eric]
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

[There is still the question of how these weights are determined. Is this the job of the third expert? Or is the third expert given these weights, and he just determines how to apply them? -Eric]

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon)

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

[I agree that a metric (or several) is crucial to our framework, but what is your demonstration of the feasibility or usability of these particular values you've chosen. On an unrelated note, we talked a lot about how some of these might be "convertible" into others…do you see these metrics as different sides of the same coin? Can the units of one be "converted" into the units of another, like gallons to liters? [[User:E J Kalafarski|E J Kalafarski]] 15:10, 29 April 2009 (UTC)]

'''review'''

* Words cannot express how amazing this contribution is. It should be given as much money as requested without question.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

'''review'''

* Cool- I like the xml. (Gideon)

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

Evaluation method:
* Evaluate an interface via the proposed framework, as well as traditional CPM-GOMS and ACT-R methods. Also have a small team of interface design experts evaluate the interface. Solicit comments on the current interface and suggestions for improvement from each team/method, and compare the accuracy and validity of the results.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-29T16:45:42Z

Eric Sodomka: /* Reconciling Usability Heuristics with Cognitive Theory */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)]

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon)

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and [...HCI design principles? -Eric]
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon)

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

[I agree that a metric (or several) is crucial to our framework, but what is your demonstration of the feasibility or usability of these particular values you've chosen. On an unrelated note, we talked a lot about how some of these might be "convertible" into others…do you see these metrics as different sides of the same coin? Can the units of one be "converted" into the units of another, like gallons to liters? [[User:E J Kalafarski|E J Kalafarski]] 15:10, 29 April 2009 (UTC)]

'''review'''

* Words cannot express how amazing this contribution is. It should be given as much money as requested without question.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

'''review'''

* Cool- I like the xml. (Gideon)

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

Evaluation method:
* Evaluate an interface via the proposed framework, as well as traditional CPM-GOMS and ACT-R methods. Also have a small team of interface design experts evaluate the interface. Solicit comments on the current interface and suggestions for improvement from each team/method, and compare the accuracy and validity of the results.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-29T16:43:23Z

Eric Sodomka: /* Reconciling Usability Heuristics with Cognitive Theory */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies. [Is there an explicit, quantifiable demonstration you can propose? Such some kind of conclusive comparison between automated capture techniques and manual techniques? Or maybe a demonstration of some kind of methodical, but still low-level, procedure for clustering related interaction primitives? [[User:E J Kalafarski|E J Kalafarski]] 15:02, 29 April 2009 (UTC)]

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

''''review''''
* This is recording everything possible during an interaction. It is necessary to do for our project. (Gideon)

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods. [As above, how will this be validated quantitatively? Comparing two methodologies for manual chunking? What would these differing methodologies be? [[User:E J Kalafarski|E J Kalafarski]] 15:06, 29 April 2009 (UTC)]

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

'''review'''
* With what level of accuracy can this be performed. I like the idea, its worthy of a nice project in and of itself. (Gideon)

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]
[I agree that the wording is is a little confusing. The framework is not weighted, but rather it weights the different guidelines/principles. It might also be worth explaining how this is a useful contribution, e.g. does it allow for more accurate interface evaluation? -Eric]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

'''review'''

* This seems like the 2d matrix? Is this implemented as a module? (Gideon)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

'''review'''

* I see this as being the "black box" for our architecture? If so, good. Wouldn't the dependencies be any/all modules? (Gideon)

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

[I agree that a metric (or several) is crucial to our framework, but what is your demonstration of the feasibility or usability of these particular values you've chosen. On an unrelated note, we talked a lot about how some of these might be "convertible" into others…do you see these metrics as different sides of the same coin? Can the units of one be "converted" into the units of another, like gallons to liters? [[User:E J Kalafarski|E J Kalafarski]] 15:10, 29 April 2009 (UTC)]

'''review'''

* Words cannot express how amazing this contribution is. It should be given as much money as requested without question.

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

'''review'''

* What exactly are the differences between this and EJ's earlier contribution? I think that if they are the same, this one is a bit more clear, IMHO. (Gideon)
[I have a similar question to Gideon's. My hunch is that this defines a sort of "namespace" for the modules themselves, while the other contribution (the one you guys assigned to me) is more explicitly the aggregator of these outputs. Correct me if I'm wrong. But if that's the case, the two contributions might need to be validated similarly/together. [[User:E J Kalafarski|E J Kalafarski]] 15:17, 29 April 2009 (UTC)]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

'''review'''

* Cool- I like the xml. (Gideon)

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

Evaluation method:
* Evaluate an interface via the proposed framework, as well as traditional CPM-GOMS and ACT-R methods. Also have a small team of interface design experts evaluate the interface. Solicit comments on the current interface and suggestions for improvement from each team/method, and compare the accuracy and validity of the results.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*Provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.
*Integrate this module into our novel cognitive framework for interface evaluation.

'''Contributions'''
* [Not sure here. Is this really novel?] [I like it. I think it's necessary. I think we demonstrated in class that this has not been formalized and standardized already. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module by generating a formal description of an interface for scientific visualization, a formal description of a task to perform with the interface, and we will correlate the estimated time to complete the task based on Fitt's law with the actual time required based on several user traces. [Check. [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Dependencies'''
*Requires a formal description of the interface with graph nodes representing clickable interface elements and graph edges representing the physical (on-screen) distance between adjacent nodes. [Is this graph format you're suggesting part of the proposal, or is the "language" itself a dependency from literature? [[User:E J Kalafarski|E J Kalafarski]] 15:22, 29 April 2009 (UTC)]

'''Module Description'''

*Inputs
**A formal description of the interface and its elements (e.g. buttons).
**A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.
**The physical distances between interface elements along those paths.
**The width of those elements along the most likely axes of motion.
**Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

*Output
**The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

'''Aims'''
*To provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing specified tasks.

'''Contributions'''
*A quantitative measure of the extent to which an interface suggests to the user the actions that it is capable of performing.
*A quantitative, indirect measure of the extent to which an interface facilitates (or hinders) the use of fast perceptual mechanisms.
[Again, I'm a fan. I don't think this has been formalized already. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Demonstrations'''
*We will demonstrate the feasibility of this module through the following experiment:
**Specify a task for a user to perform with scientific visualization software.
**There should be several different ways to complete the task (paths through the space of possible interface actions).
**Some of these paths will be more direct than others.
**We will then measure the number of task-relevant affordances that were perceived and acted upon by analyzing the user trace, and the time required to complete the task.
**Use the formula: (affordances perceived) / [(relevant affordances present) * (time to complete task)].
**Correlate the resulting scores with verbal reports on naturalness and ease-of-use for the interface.

'''Dependencies'''
*Requires a method for capturing user traces that include formalized records of interface elements used for a particular, pre-specified task or set of tasks.
*Providing suggestions/recommendations will require interaction with other modules that analyze the perceptual salience of interface elements.
[Some kind of formalized list or arbitrary classification of affordances might be necessary to limit the scope of this contribution. [[User:E J Kalafarski|E J Kalafarski]] 15:24, 29 April 2009 (UTC)]

'''Description of the Module'''

*Module Inputs''
**Formalized descriptions of...
***Interface elements
***Their associated actions
***The functions of those actions
***A particular task
***User traces for that task

*Processing
**Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

*Output
**The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-28T20:44:26Z

Eric Sodomka: /* Parallel Framework for Evaluation Modules */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies.

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods.

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Final contributions

2009-04-28T20:44:03Z

Eric Sodomka: /* Contributions */

= Specific Aims and Contributions (to be separated later) =

[David: I have removed some unattributed stuff that wasn't in the format described in assignment 13. If that was in error, go back to an earlier version to recover the text, attribute it, and put in the correct format.]

== Recording User-Interaction Primitives ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multi-modal, HCI framework for recording rich interaction-history data in existing applications.

'''Demonstration'''
* Techniques for pupil-tracking, auditory-recognition, and muscle-activity monitoring will be evaluated with respect to accuracy, sampling rate, and computational cost in a series of quantitative studies.

'''Dependencies'''
* Hardware for pupil-tracking and muscle activity monitoring. Commercial software packages for such software may be required.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==

* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic-chunking techniques for interaction.

'''Demonstration'''
* These techniques will be validated quantitatively with respect to efficiency and user error rate on a set of timed-tasks over a series of case studies. In addition, qualitative feedback on user satisfaction will be incorporated in the evaluation of our methods.

'''Dependencies'''
* Software framework for collecting user interactions.
* Formal description of interface functionality.
* Description of data objects that can be manipulated through interaction.

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contribution''': A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

[David: I'm having trouble parsing this contribution. How do you weight a framework? Also, I'd like to have a little more sense of how this might fit into the bigger picture. The assignment didn't ask for that, but is there some way to provide some of that context?]

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependency''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

'''Contributions''': Using a limited set of interface evaluation modules for analysis, we demonstrate, in a narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

[David: I'm a little lost. Can you make this long sentence into a couple shorter simpler ones? I think I can imagine what you are getting at, but I'm not sure.]

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

[David: nice demonstration!]

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

[David: I think this also has, as a dependency, some kind of framework for the modules. "narrow but comprehensive" sounds challenging. ]

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

'''Contribution:''' 'Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.

'''Demonstration:''' Run a series of user studies and compare users' performance to expected performance, as given by the following interface evaluation methods:
# Traditional, manual interface evaluation
#* As a baseline.
# Using our system with a single module
#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
#* For validating the use of a dynamic weighting system.
# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
#* For validating the use of weighting across multiple utility dimensions.

'''Dependencies:''' Requires a good set of modules to plug into the framework.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

[David: I think that the work proposed here is interesting, but I'm not sure that it is organized by "contribution". Try labeling the contributions and the demonstrations explicitly. A "tool" is usually not a contribution -- an evaluation of the tool could be. If this distinction isn't clear, let's talk. For these particular contributions, phrasing and integrating them is a challenge because some of the contributions are demonstrations of other contributions...]

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

=== Anti-Pattern Conflict Resolution ===
* Owner: [[User: Ian Spector | Ian Spector]]

Interface design patterns are defined as reusable elements which provide solutions to common problems. For instance, we expect that an arrow in the top left corner of a window which points to the left, when clicked upon, will take the user to a previous screen. Furthermore, we expect that buttons in the top left corner of a window will in some way relate to navigation.

An anti-pattern is a design pattern which breaks standard design convention, creating more problems than it solves. An example of an anti-pattern would be if in place of the 'back' button on your web browser, there would be a 'view history' button.

The purpose of this module would be to analyze an interface to see if any anti-patterns exist, identify where they are in the interface, and then suggest alternatives.

'''Inputs'''
*Formal interface description
*Tasks which can be performed within the interface
*A library of standard design patterns
*Outputs from the 'Affordances' module
*Uncommon / Custom additional pattern library (optional)

'''Outputs'''
*Identification of interface elements whose placement or function are contrary to the pattern library
*Recommendations for alternative functionality or placement of such elements.

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an interface and at what stages our proposal fits in and how.''

The interface design process is critical to the creation of a quality end product. The process of creating an interface can also be used as a model for analyzing a finished one.

There are a number of different philosophies on how to best design software and in turn, interface. Currently, agile development using an incremental process such as [http://en.wikipedia.org/wiki/SCRUM Scrum] has become a well known and generally practiced procedure.

The steps to create interfaces varies significantly from text to text, although the [http://http://cfg.cit.cornell.edu/design/process.html Common Front Group at Cornell] has succinctly been able to reduce this variety into six simple steps:

*Requirement Sketching
*Conceptual Design
*Logical Design
*Physical Design
*Construction
*Usability Testing

This can be broken down further into just information architecture design followed by physical design and testing.

As far as this proposal is concerned in the context of interface design, the goal of our proposal is to improve what is involved with later end of the middle two portions: logical and physical design. Prior to feeding an interface to the system we are proposing, designers should have already created a baseline model for review that should exhibit the majority, if not all, of the functionality listed in the interface requirements. Once this initial interface has been created, our system will aid in rapidly iterating through the physical design process. The ultimate end products are then subject to human usability testing.

CS295J/Research proposal (draft 2)

2009-04-28T16:06:49Z

Eric Sodomka: /* Contributions */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref name = bob/> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008).

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependence''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

Using a limited set of interface evaluation modules for analysis, we demonstrate, in narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its modules taken in isolation.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.
*Dependencies: Having a good set of modules to plug into the framework.

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Assignment 13'''

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-28T16:06:29Z

Eric Sodomka: /* Contributions */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref name = bob/> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008).

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

A weighted framework for the unification of established heuristic usability guidelines and accepted cognitive principles.

'''Demonstration''': Three groups of experts anecdotally apply cognitive principles, heuristic usability principles, and a combination of the two.
* A "cognition expert," given a constrained, limited-functionality interface, develops an independent evaluative value for each interface element based on accepted cognitive principles.
* A "usability expert" develops an independent evaluative value for each interface element based on accepted heuristic guidelines.
* A third expert applies several unified cognitive analogues from a matrix of weighted cognitive and
* User testing demonstrates the assumed efficacy and applicability of the matricized analogues versus independent application of analogued principles.

'''Dependency''': A set of established cognitive principles, selected with an eye toward heuristic analogues.

'''Dependence''': A set of established heuristic design guidelines, selected with an eye toward cognitive analogues.

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

Using a limited set of interface evaluation modules for analysis, we demonstrate, in narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Create a framework that provides better interface evaluations than currently existing techniques, and a module weighting system that provides better evaluations than any of its individual modules.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.
*Dependencies: Having a good set of modules to plug into the framework.

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Assignment 13'''

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-28T16:04:35Z

Eric Sodomka: /* Contributions */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref name = bob/> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008).

== Reconciling Usability Heuristics with Cognitive Theory ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

== Evaluation for Recommendation and Incremental Improvement ==
* Owner: [[User:E J Kalafarski|E J Kalafarski]] 14:56, 28 April 2009 (UTC)

Using a limited set of interface evaluation modules for analysis, we demonstrate, in narrow and controlled manner, the proposed efficiency and accuracy of a method of aggregating individual interface suggestions based on accepted CPM principles (e.g. Fitts' Law and Affordance) and applying them to the incremental improvement of the interface.

'''Demonstration''': A narrow but comprehensive study to demonstrate the efficacy and efficiency of automated aggregation of interface evaluation versus independent, analysis. Shows not total replacement, but a gain in speed of evaluation comparable to the loss of control and feature.
* Given a carefully-constrained interface, perhaps with as few as two buttons and a minimalist feature set, expert interface designer given the individual results of several basic evaluation modules makes recommendations and suggestions to the design of the interface.
* Aggregation meta-module conducts similar survey of module outputs, outputting recommendations and suggestions for improvement of given interface.
* Separate independent body of experts then implements the two sets of suggestions and performs user study on the resultant interfaces, analyzing usability change and comparing it to the time and resources committed by the evaluation expert and the aggregation module, respectively.

'''Dependency''': Module for the analysis of Fitts' Law as it applies to the individual elements of a given interface.

'''Dependency''': Module for the analysis of a Affordance as it applies to the individual elements of a given interface.

== Evaluation Metrics ==
Owner: [[User:Gideon Goldin | Gideon Goldin]]

:*Architecture Outputs
::* Time (time to complete task)
:::# Performs as well or better than CPM-GOMS, demonstrated with user tasks
:::* Dependencies
:::# CPM-GOMS Module with cognitive load extension
::* Cognitive Load
:::# Predicts cognitive load during tasks, demonstrated with user tasks
:::* Dependencies
::::# CPM-GOMS Module with cognitive load extension
::::# Facial Gesture Recognition Module
::* Frustration
:::# Accurately predicts users' frustration levels, demonstrated with user tasks
:::* Dependencies
::::# Facial Gesture Recognition Module
::::# Galvanic Skin Response Module
::::# Interface Efficiency Module
::* Aesthetic Appeal
:::# Analyzes if the interface is aesthetically unpleasing, demonstrated with user tasks
:::* Dependencies
::::# Aesthetics Module
::* Simplicity
:::# Analyzes how simple the interface is, demonstrated with user tasks
:::* Dependencies
::::# Interface Efficiency Module

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Create a framework that provides better interface evaluations than any of its individual modules.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.
*Dependencies: Having a good set of modules to plug into the framework.

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Assignment 13'''

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-28T13:35:25Z

Eric Sodomka: /* Contributions */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref name = bob/> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008).

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Create a framework that provides better interface evaluations than any of its individual modules.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Assignment 13'''

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-28T13:35:04Z

Eric Sodomka: /* Contributions */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref name = bob/> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008).

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Created a framework that provides better interface evaluations than any of its individual modules.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Assignment 13'''

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-28T13:12:51Z

Eric Sodomka: /* Contributions */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref name = bob/> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008).

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Created a framework that provides better interface evaluations than any of its individual components.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Assignment 13'''

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-28T13:11:05Z

Eric Sodomka: /* Parallel Framework for Evaluation Modules */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]], [[User: Trevor O'Brien | Trevor]]

'''Propose:''' The design, application and evaluation of a novel, cognition-based, computational framework for assessing interface design and providing automated suggestions to optimize usability.

'''Evaluation Methodology:''' Our techniques will be evaluated quantitatively through a series of user-study trials, as well as qualitatively by a team of expert interface designers.

'''Contributions and Significance:''' We expect this work to make the following contributions:
# design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
# design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
# an extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
# specification (language?) of how to define an interface evaluation module and how to integrate it into a larger system.
# (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

== Cognitive Models ==
I plan to port over most of the background on cognitive models of HCI from the old proposal

Additions will comprise of:
*CPM-GOMS as a bridge from GOMS architecture to the promising procedural optimization of the Model Human Processor
**Context of CPM development, discuss its relation to original GOMS and KLM
***Establish the tasks which were relevant for optimization when CPM was developed and note that its obsolescence may have been unavoidable
**Focus on CPM as the first step in transitioning from descriptive data, provided by mounting efforts in the cognitive sciences realm to discover the nature of task processing and accomplishment, to prescriptive algorithms which can predict an interface’s efficiency and suggest improvements
**CPM’s purpose as an abstraction of cognitive processing – a symbolic representation not designed for accuracy but precision
**CPM’s successful trials, e.g. Ernestine
***Implications of this project include CPM’s ability to accurately estimate processing at a psychomotor level
***Project does suggest limitations, however, when one attempts to examine more complex tasks which involve deeper and more numerous cognitive processes
*ACT-R as an example of a progressive cognitive modeling tool
**A tool clearly built by and for cognitive scientists, and as a result presents a much more accurate view of human processing – helpful for our research
**Built-in automation, which now seems to be a standard feature of cognitive modeling tools
**Still an abstraction of cognitive processing, but makes adaptation to cutting-edge cognitive research findings an integral aspect of its modular structure
**Expand on its focus on multi-tasking, taking what was a huge advance between GOMS and its CPM variation and bringing the simulation several steps closer to approximating the nature of cognition in regards to HCI
**Far more accessible both for researchers and the lay user/designer in its portability to LISP, pre-construction of modules representing cognitive capacities and underlying algorithms modeling paths of cognitive processing

==Design guidelines==
A multitude of rule sets exist for the design of not only interface, but architecture, city planning, and software development. They can range in scale from one primary rule to as many Christopher Alexander's 253 rules for urban environments,<ref>[http://hci.rwth-aachen.de/materials/publications/borchers2000a.pdf Borchers, Jan O. "A Pattern Approach to Interaction Design." 2000.]</ref> which he introduced with the concept design patterns in the 1970s. Study has likewise been conducted on the use of these rules:<ref>http://stl.cs.queensu.ca/~graham/cisc836/lectures/readings/tetzlaff-guidelines.pdf</ref> guidelines are often only partially understood, indistinct to the developer, and "fraught" with potential usability problems given a real-world situation.

===Application to AUE===
And yet, the vast majority of guideline sets, including the most popular rulesets, have been arrived at heuristically. The most successful, such as Raskin's and Schneiderman's, have been forged from years of observation instead of empirical study and experimentation. The problem is similar to the problem of circular logic faced by automated usability evaluations: an automated system is limited in the suggestions it can offer to a set of preprogrammed guidelines which have often not been subjected to rigorous experimentation.<ref>[http://www.eecs.berkeley.edu/Pubs/TechRpts/2000/CSD-00-1105.pdf Ivory, M and Hearst, M. "The State of the Art in Automated Usability Evaluation of User Interfaces." ACM Computing Surveys (CSUR), 2001.]</ref> In the vast majority of existing studies, emphasis has traditionally been placed on either the development of guidelines or the application of existing guidelines to automated evaluation. A mutually-reinforcing development of both simultaneously has not been attempted.

Overlap between rulesets is inevitable and unavoidable. For our purposes of evaluating existing rulesets efficiently, without extracting and analyzing each rule individually, it may be desirable to identify the the overarching ''principles'' or ''philosophy'' (max. 2 or 3) for a given ruleset and determining their quantitative relevance to problems of cognition.

===Popular and seminal examples===
Schneiderman's [http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html Eight Golden Rules] date to 1987 and are arguably the most-cited. They are heuristic, but can be somewhat classified by cognitive objective: at least two rules apply primarily to ''repeated use'', versus ''discoverability''. Up to five of Schneiderman's rules emphasize ''predictability'' in the outcomes of operations and ''increased feedback and control'' in the agency of the user. His final rule, paradoxically, removes control from the user by suggesting a reduced short-term memory load, which we can arguably classify as ''simplicity''.

Raskin's [http://www.mprove.de/script/02/raskin/designrules.html Design Rules] are classified into five principles by the author, augmented by definitions and supporting rules. While one principle is primarily aesthetic (a design problem arguably out of the bounds of this proposal) and one is a basic endorsement of testing, the remaining three begin to reflect philosophies similar to Schneiderman's: reliability or ''predictability'', ''simplicity'' or ''efficiency'' (which we can construe as two sides of the same coin), and finally he introduces a concept of ''uninterruptibility''.

Maeda's [http://lawsofsimplicity.com/?cat=5&order=ASC Laws of Simplicity] are fewer, and ostensibly emphasize ''simplicity'' exclusively, although elements of ''use'' as related by Schneiderman's rules and ''efficiency'' as defined by Raskin may be facets of this simplicity. Google's corporate mission statement presents [http://www.google.com/corporate/ux.html Ten Principles], only half of which can be considered true interface guidelines. ''Efficiency'' and ''simplicity'' are cited explicitly, aesthetics are once again noted as crucial, and working within a user's trust is another application of ''predictability''.

===Elements and goals of a guideline set===
Myriad rulesets exist, but variation becomes scarce—it indeed seems possible to parse these common rulesets into overarching principles that can be converted to or associated with quantifiable cognitive properties. For example, it is likely ''simplicity'' has an analogue in the short-term memory retention or visual retention of the user, vis a vis the rule of [http://books.google.com/books?hl=en&lr=&id=j5q0VvOGExYC&oi=fnd&pg=PA357&dq=seven+plus+or+minus+two&ots=prI3PKJBar&sig=vOZnqpnkXKGYWxK6_XlA4I_CRyI Seven, Plus or Minus Two]. ''Predictability'' likewise may have an analogue in Activity Theory, in regards to a user's perceptual expectations for a given action; ''uninterruptibility'' has implications in cognitive task-switching;<ref>[http://portal.acm.org/citation.cfm?id=985692.985715&coll=Portal&dl=ACM&CFID=21136843&CFTOKEN=23841774 Czerwinski, Horvitz, and White. "A Diary Study of Task Switching and Interruptions." Proceedings of the SIGCHI conference on Human factors in computing systems, 2004.]</ref> and so forth.

Within the scope of this proposal, we aim to reduce and refine these philosophies found in seminal rulesets and identify their logical cognitive analogues. By assigning a quantifiable taxonomy to these principles, we will be able to rank and weight them with regard to their real-world applicability, developing a set of "meta-guidelines" and rules for applying them to a given interface in an automated manner. Combined with cognitive models and multi-modal HCI analysis, we seek to develop, in parallel with these guidelines, the interface evaluation system responsible for their application.

== Perception and Action (in progress) ==

*Information Processing Approach
**Advantages
***Formalism eases translation of theory into scripting language
**Disadvantages
***Assumes symbolic representation

*Ecological (Gibsonian) Approach
**Advantages
***Emphasis on bodily and environmental constraints
**Disadvantages
***Lack of formalism hinders translation of theory into scripting language

= Specific Aims and Contributions (to be separated later) =

'''Specific Aims'''
* Incorporate interaction history mechanisms into a set of existing applications.
* Perform user-study evaluation of history-collection techniques.
* Distill a set of cognitive principles/models, and evaluate empirically?
* Build/buy sensing system to include pupil-tracking, muscle-activity monitoring, auditory recognition.
* Design techniques for manual/semi-automated/automated construction of <insert favorite cognitive model here> from interaction histories and sensing data.
* Design system for posterior analysis of interaction history w.r.t. <insert favorite cognitive model here>, evaluating critical path <or equivalent> trajectories.
* Design cognition-based techniques for detecting bottlenecks in critical paths, and offering optimized alternatives.
* Perform quantitative user-study evaluations, collect qualitative feedback from expert interface designers.

'''Contributions'''

* Design-space analysis and quantitative evaluation of cognition-based techniques for assessing user interfaces.
* Design and quantitative evaluation of techniques for suggesting optimized interface-design changes.
* An extensible, multimodal software architecture for capturing user traces integrated with pupil-tracking data, auditory recognition, and muscle-activity monitoring.
* (there may be more here, like testing different cognitive models, generating a markup language to represent interfaces, maybe even a unique metric space for interface usability)

--

See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Recording User-Interaction Primitives ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop system for logging low-level user-interactions within existing applications. By low-level interactions, we refer to those interactions for which a quantitative, predictive, GOMS-like model may be generated.
* At the same scale of granularity, integrate explicit interface-interactions with multimodal-sensing data. i.e. pupil-tracking, muscle-activity monitoring, auditory recognition, EEG data, and facial and posture-focused video.

'''Contributions'''
* Extensible, multimodal, HCI framework for recording rich interaction-history data in existing applications.

'''Significance'''
* Provide rich history data to serve as the basis for novel quantitative interface evaluation.
* Aid expert HCI and visual designers in traditional design processes.
* Provide data for automated machine-learning strategies applied to interaction.

'''Background and Related Work'''
* The utility of interaction histories with respect to assessing interface design has been demonstrated in <ref name = bob> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008) </ref>.
* In addition, data management histories have been shown effective in the visualization community in <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-MED.pdf Callahan-2006-MED] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Callahan-2006-VVM.pdf Callahan-2006-VVM] </ref> <ref> [http://www.cs.brown.edu/people/trevor/Papers/Bavoil-2005-VEI.pdf Bavoil-2005-VEI]</ref>, providing visualizations by analogy <ref> [http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4376187&isnumber=4376125 Querying and Creating Visualizations by Analogy] </ref> and offering automated suggestions <ref> [http://www.cs.utah.edu/~juliana/pub/tvcg-recommendation2008.pdf VisComplete: Automating Suggestions from Visualization Pipelines] </ref>, which we expect to generalize to user interaction history.

== Semantic-level Interaction Chunking ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

'''Aims'''
* Develop techniques for ''chunking'' low-level interaction primitives into ''semantic-level'' interactions, given an application's functionality and data-context. (And de-chunking? Invertible mapping needed?)
* Perform user-study evaluation to validate chunking methods.

'''Contributions'''
* Design and user-study evaluation of semantic chunking techniques.

'''Significance'''
* Low-level <--> semantic-level mapping allows for cognitive-modeling to be applied at a functionality level, where low-level interaction techniques can be swapped out. This will allow our interface assessment system to make feasible suggestions for more optimal interface design.

'''Background and Related Work'''
* Chunking interactions has been studied in the HCI community as in <ref name = bob/> [http://www.cs.brown.edu/people/trevor/Papers/Heer-2008-GraphicalHistories.pdf Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation] (InfoVis 2008).

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

==== Contributions ====

* '''Created a framework that provides better interface evaluations than any of its individual components.'''

* Demonstrate by running user studies and comparing this performance to expected performance, as given by the following interface evaluation methods:
*# Traditional, manual interface evaluation
*#* As a baseline.
*# Using our system with a single module
*#* "Are any of our individual modules better than currently existing methods of interface evaluation?".
*# Using our system with multiple modules, but have aggregator give a fixed, equal weighting to each module
*#* As a baseline for our aggregator: want to show that the value of adding the dynamic weighting.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, but have each module get a single weighting for all dimensions (time, fatigue, etc.)
*#* For validating the use of a dynamic weighting system.
*# Using our system with multiple modules, and allow the aggregator to adjust weightings for each module, and allow the module to give different weightings to every dimension of the module (time, fatigue, etc.)
*#* For validating the use of weighting across multiple utility dimensions.

==== Module Inputs (Incomplete) ====
* A set of utility dimensions {d1, d2, ...} are defined in the framework. These could be {d1="time", d2="fatigue", ...}
* A set of interaction functions. These specify all of the information that the application wants to give users or get from them. It is not tied to a specific interface. For example, the fact that an applications shows videos would be included here. Whether it displays them embedded, in a pop-up window or fullscreen would not.
* A mapping of interaction functions to interface elements (e.g., buttons, windows, dials,...). Lots of optional information describing visual properties, associated text, physical interactions (e.g., turning the dial clockwise increases the input value) and timing.
* Functional user traces - sequences of interaction functions that represent typical user interactions with the application. Could include a goal hierarchy, in which case the function sequence is at the bottom of the hierarchy.

==== Module Outputs ====
* Every module ouputs at least one of the following:
** An evaluation of the interface
*** This can be on any or all of the utility dimensions, e.g. evaluation={d1=score1, d2=score2, ...}
*** This can alternately be an overall evaluation, ignoring dimensions, e.g. evaluation={score}
**** In this case, the aggregator will treat this as the module giving the same score to all dimensions. Which dimension this evaluator actually predicts well on can be learned by the aggregator over time.
** Recommendation(s) for improving the interface
***This can be a textual description of what changes the designer should make
***This can alternately be a transformation that can automatically be applied to the interface language (without designer intervention)
***In addition to the textual or transformational description of the recommendation, a "change in evaluation" is output to describe how specifically the value will improve the interface
****Recommendation = {description="make this change", Δevaluation={d1=score1, d2=score2, ...}
****Like before, this Δevaluation can cover any number of dimensions, or it can be generic.
***Either a single recommendation or a set of recommendations can be output

==== Aggregator Inputs ====
The aggregator receives as input the outputs of all the modules.

==== Aggregator Outputs ====

Outputs for the aggregator are the same as the outputs for each module. The difference is that the aggregator will consider all the module outputs, and arrive at a merged output based on the past performance of each of the modules.

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

A "meta-module" called the aggregator will be responsible for assembling and formatting the output of all other modules into a structure that is both extensible and immediately usable, by both an automated designer or a human designer.

===Requirements===
The aggregator's functionality, then, is defined by its '''inputs''', the outputs of the other modules, and the desired '''output''' of the system as a whole, per its position in the architecture. Its purpose is largely formatting and reconciliation of the products of the multitudinous (and extensible) modules. The output of the aggregator must meet several requirements: first, to generate a set of human-readable suggestions for the improvement of the given interface; second, to generate a machine-readable, but also analyzable, evaluation of the various characteristics of the interface and accompanying user traces.

From these specifications, it is logical to assume that a common language or format will be required for the output of individual modules. We propose an XML-based file format, allowing: (1) a section for the standardized identification of problem areas, applicable rules, and proposed improvements, generalized by the individual module and mapped to a single element, or group of elements, in the original interface specification; (2) a section for specification of generalizable "utility" functions, allowing a module to specify how much a measurable quantity of utility is positively or negatively affected by properties of the input interface; (3) new, user-definable sections for evaluations of the given interface not covered by the first two sections. The first two sections are capable of conveying the vast majority of module outputs predicted at this time, but the XML can extensibly allow modules to pass on whatever information may become prominent in the future.

===Specification===
<module id="Fitts-Law">
<interface-elements>
<element>
<desc>submit button</desc>
<problem>
<desc>size</desc>
<suggestion>width *= 2</suggestion>
<suggestion>height *= 2</suggestion>
<human-suggestion>Increase size relative to other elements</human-suggestion>
</problem>
</element>
</interface-elements>
<utility>
<dimension>
<desc>time</desc>
<value>0:15:35</value>
</dimension>
<dimension>
<desc>frustration</desc>
<value>pulling hair out</value>
</dimension>
<dimension>
<desc>efficiency</desc>
<value>13.2s/KPM task</value>
<value>0.56m/CPM task</value>
</dimension>
</utility>
<tasks>
<task>
<desc>complete form</desc>
</task>
<task>
<desc>lookup SSN</desc>
</task>
<task>
<desc>format phone number</desc>
</task>
</tasks>
</module>

===Logic===
This file provided by each module is then the input for the aggregator. The aggregator's most straightforward function is the compilation of the "problem areas," assembling them and noting problem areas and suggestions that are recommended by more than one module, and weighting them accordingly in its final report. These weightings can begin in an equal state, but the aggregator should be capable of learning iteratively which modules' results are most relevant to the user and update weightings accordingly. This may need to be accomplished with manual tuning, or a machine-learning algorithm capable of determining which modules most often agree with others.

Secondly, the aggregator compiles the utility functions provided by the module specs. This, again, is a summation of similarly-described values from the various modules.

When confronted with user-defined sections of the XML spec, the aggregator is primarily responsible for compiling them and sending them along to the output of the machine. Even if the aggregator does not recognize a section or property of the evaluative spec, if it sees the property reported by more than one module it should be capable of aggregating these intelligently. In future versions of the spec, it should be possible for a module to provide instructions for the aggregator on how to handle unrecognized sections of the XML.

From these compilations, then, the aggregator should be capable of outputting both aggregated human-readable suggestions on interface improvements for a human designer, as well as a comprehensive evaluation of the interface's effectiveness at the given task traces. Again, this is dependent on the specification of the system as a whole, but is likely to include measures and comparisons, graphings of task versus utility, and quantitative measures of an element's effectiveness.

''This section is necessarily defined by the output of the individual modules (which I already expect to be of varied and arbitrary structure) and the desired output of the machine as a whole. It will likely need to be revised heavily after other modules and the "Parellel Framework" section are defined.'' [[User:E J Kalafarski|E J Kalafarski]] 12:34, 24 April 2009 (UTC)

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

I’m hoping to have some input on this section, because it seems to be the crux of the “black box” into which we take the inputs of interface description, user traces, etc. and get our outputs (time, recommendations, etc.). I know at least a few people have pretty strong thoughts on the matter and we ought to discuss the final structure.

That said – my proposal for the module:
*In my opinion the concept of the Model Human Processor (at least as applied in CPM) is outdated – it’s too economic/overly parsimonious in its conception of human activity. I think we need to create a structure which accounts for more realistic conditions of HCI including multitasking, aspects of distributed cognition (and other relevant uses of tools – as far as I can tell CPM doesn’t take into account any sort of productivity aids), executive control processes of attention, etc. ACT-R appears to take steps towards this but we would probably need to look at their algorithms to know for sure.
*Critical paths will continue to play an important role – we should in fact emphasize that part of this tool’s purpose will be a description not only of ways in which the interface should be modified to best fit a critical path, but also ways in which the user’s ought to be instructed in their use of the path. This feedback mechanism could be bidirectional – if the model’s predictions of the user’s goals are incorrect, the critical path determined will also be incorrect and the interface inherently suboptimal. The user could be prompted with a tooltip explaining in brief why and how the interface has changed, along with options to revert, select other configurations (euphemized by goals), and to view a short video detailing how to properly use the interface.
*Call me crazy but, if we assume designers will be willing to code a model of their interfaces into our ACT-R-esque language, could we allow that model to be fairly transparent to the user, who could use a gui to input their goals to find an analogue in the program which would subsequently rearrange its interface to fit the user’s needs? Even if not useful to the users, such dynamic modeling could really help designers (IMO)
*I think the model should do its best to accept models written for ACT-R and whatever other cognitive models there are out there – gives us the best chance of early adoption
*I would particularly appreciate input on the number/complexity/type of inputs we’ll be using, as well as the same qualities for the output.

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

Schneiderman's Eight Golden Rules and Jakob Nielsen's Ten Heuristics are perhaps the most famous and well-regarded heuristic design guidelines to emerge over the last twenty years. Although the explicit theoretical basis for such heuristics is controversial and not well-explored, the empirical success of these guidelines is established and accepted. This module will parse out up to three or four common (that is, intersecting) principles from these accepted guidelines and apply them to the input interface.

As an example, we identify an analogous principle that appears in Schneiderman ("Reduce short-term memory load")<ref>http://faculty.washington.edu/jtenenbg/courses/360/f04/sessions/schneidermanGoldenRules.html</ref> and Nielsen ("Recognition rather than recall/Minimize the user's memory load")<ref>http://www.useit.com/papers/heuristic/heuristic_list.html</ref>. The input interface is then evaluated for the consideration of the principle, based on an explicit formal description of the interface, such as XAML or XUL. The module attempts to determine how effectively the interface demonstrates the principle. When analyzing an interface for several principles that may be conflicting or opposing in a given context, the module makes use of a hard-coded but iterative (and evolving) weighting of these principles, based on (1) how often they appear in the training set of accepted sets of guidelines, (2) how analogues a heuristic principle is to a cognitive principle in a parallel training set, and (3) how effective the principle's associated suggestion is found to be using a feedback mechanism.

'''Inputs''' 
# A formal description of the interface and its elements (e.g. buttons).
# A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

'''Output''' 
Standard XML-formatted file containing problem areas of the input interface, suggestions for each problem area based on principles that were found to have a strong application to a problem element and the problem itself, and a human-readable generated analysis of the element's affinity for the principle. Quantitative outputs will not be possible based on heuristic guidelines, and the "utility" section of this module's output is likely to be blank.

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This module provides an estimate of the required time to complete various tasks that have been decomposed into formalized sequences of interactions with interface elements, and will provide evaluations and recommendations for optimizing the time required to complete those tasks using the interface.''

'''Inputs'''

1. A formal description of the interface and its elements (e.g. buttons).

2. A formal description of a particular task and the possible paths through a subset of interface elements that permit the user to accomplish that task.

3. The physical distances between interface elements along those paths.

4. The width of those elements along the most likely axes of motion.

5. Device (e.g. mouse) characteristics including start/stop time and the inherent speed limitations of the device.

'''Output'''

The module will then use the Shannon formulation of Fitt's Law to compute the average time needed to complete the task along those paths.

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on a measure of the extent to which the user perceives the relevant affordances of the interface when performing a number of specified tasks.''

'''Inputs'''

Formalized descriptions of...

1. Interface elements

2. Their associated actions

3. The functions of those actions

4. A particular task

5. User traces for that task.

Inputs (1-4) are then used to generate a "user-independent" space of possible functions that the interface is capable of performing with respect to a given task -- what the interface "affords" the user. From this set of possible interactions, our model will then determine the subset of optimal paths for performing a particular task. The user trace (5) is then used to determine what functions actually were performed in the course of a given task of interest and this information is then compared to the optimal path data to determine the extent to which affordances of the interface are present but not perceived.

'''Output'''

The output of this module is a simple ratio of (affordances perceived) / [(relevant affordances present) * (time to complete task)] which provides a quantitative measure of the extent to which the interface is "natural" to use for a particular task.

=== Workflow, Multi-tasking and Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

'''Assignment 13'''

::* Scientific Study of Multi-tasking Workflow and the Impact of Interruptions
:::# We will undertake detailed studies to help understand the following questions:
::::# How does the size of a user's working set impact interruption resumption time?
::::# How does the size of a user's working set, when used for rapid multi-tasking, impact performance metrics?
::::# How does a user interface which supports multiple simultaneous working sets benefit interruption resumption time?
:::* No Dependencies
::* Meta-work Assistance Tool
:::# We will perform a series of ecologically-valid studies to compare user performance between a state of the art task management system (control group) and our meta-work assistance tool (experimental group)
:::* Dependent on core study completion, as some of the specific design decisions will be driven by the results of this study. However, it is worth pointing out that this separate contribution can be researched in parallel to the core study.
::* Baseline Comparison Between Module-based Model of HCI and Core Multi-tasking Study
:::# We will compare the results of the above-mentioned study in multi-tasking against results predicted by the module-based model of HCI in this proposal; this will give us an important baseline comparison, particularly given that multi-tasking and interruption involve higher brain functioning and are therefore likely difficult to predict
:::* Dependent on core study completion, as well as most of the rest of the proposal being completed to the point of being testable

Text for Assignment 12:

Add text here about how this can be used to evaluate automatic framework

There are, at least, two levels at which users work ([http://portal.acm.org/citation.cfm?id=985692.985707&coll=portal&dl=ACM&CFID=20781736&CFTOKEN=83176621 Gonzales, et al., 2004]). Users accomplish individual low-level tasks which are part of larger ''working spheres''; for example, an office worker might send several emails, create several Post-It (TM) note reminders, and then edit a word document, each of these smaller tasks being part of a single larger working sphere of "adding a new section to the website." Thus, it is important to understand this larger workflow context - which often involves extensive levels of multi-tasking, as well as switching between a variety of computing devices and traditional tools, such as notebooks. In this study it was found that the information workers surveyed typically switch individual tasks every 2 minutes and have many simultaneous working spheres which they switch between, on average every 12 minutes. This frenzied pace of switching tasks and switching working spheres suggests that users will not be using a single application or device for a long period of time, and that affordances to support this characteristic pattern of information work are important.

The purpose of this module is to integrate existing work on multi-tasking, interruption and higher-level workflow into a framework which can predict user recovery times from interruptions. Specifically, the goals of this framework will be to:

* Understand the role of the larger workflow context in user interfaces
* Understand the impact of interruptions on user workflow
* Understand how to design software which fits into the larger working spheres in which information work takes place

It is important to point out that because workflow and multi-tasking rely heavily on higher-level brain functioning, it is unrealistic within the scope of this grant to propose a system which can predict user performance given a description of a set of arbitrary software programs. Therefore, we believe this module will function much more in a qualitative role to provide context to the rest of the model. Specifically, our findings related to interruption and multi-tasking will advance the basic research question of "how do you users react to interruptions when using working sets of varying sizes?". This core HCI contribution will help to inform the rest of the outputs of the model in a qualitative manner.

'''Inputs'''

N/A

'''Outputs'''

N/A

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

* Models
** Beddeley's Model of Working Memory<ref>http://en.wikipedia.org/wiki/Baddeley%27s_model_of_working_memory</ref>
*** Episodic Buffer
** George Miller's "The magic number 7 plus or minus 2"<ref>http://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two</ref>
** The magic number 4<ref>Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24, 87-185.</ref>
* Chunking<ref>http://en.wikipedia.org/wiki/Chunking_(psychology)</ref>
* Priming<ref>http://en.wikipedia.org/wiki/Priming_(psychology)</ref>
* Subitizing and Counting<ref>http://en.wikipedia.org/wiki/Subitizing</ref>

'''Inputs'''
* Visual Stimuli
* Audio Stimuli
'''Outputs'''
* Remembered percepts
* Half-Life of percepts

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

* Learning<ref>http://en.wikipedia.org/wiki/Learning#Mathematical_models_of_learning</ref>
** Logan's instance theory of automatization<ref>http://74.125.95.132/search?q=cache:IZ-Zccsu3SEJ:psych.wisc.edu/ugstudies/psych733/logan_1988.pdf+logan+isntance+teory&cd=1&hl=en&ct=clnk&gl=us&client=firefox-a</ref>
* Fluency
** As meta-cognitive information <ref>http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VH9-4SM7PFK-4&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=10cd279fa80958981fcc3c06684c09af</ref>
** As a cognitive ''heuristic''<ref>http://en.wikipedia.org/wiki/Fluency_heuristic</ref>

'''Inputs'''
* Interface
* User goals

'''Outputs'''
* Learning curve

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

===Workflow, Multi-tasking, and Interruption===

====I. '''Goals'''====

The goals of the preliminary work are to gain qualitative insight into how information workers practice metawork, and to determine whether people might be better-supported with software which facillitates metawork and interruptions. Thus, the preliminary work should investigate, and demonstrate, the need and impact of the core goals of the project.

====II. '''Methodology'''====

Seven information workers, ages 20-38 (5 male, 2 female), were interviewed to determine which methods they use to "stay organized". An initial list of metawork strategies was established from two pilot interviews, and then a final list was compiled. Participants then responded to a series of 17 questions designed to gain insight into their metawork strategies and process. In addition, verbal interviews were conducted to get additional open-ended feedback.

====III. '''Final Results'''====
A histogram of methods people use to "stay organized" in terms of tracking things they need to do (TODOs), appointments and meetings, etc. is shown in the figure below.

[[Image:AcbGraph.jpg]]

In addition to these methods, participants also used a number of other methods, including:

* iCal
* Notes written in xterms
* "Inbox zero" method of email organization
* iGoogle Notepad (for tasks)
* Tag emails as "TODO", "Important", etc.
* Things (Organizer Software)
* Physical items placed to "remind me of things"
* Sometimes arranging windows on desk
* Keeping browser tabs open
* Bookmarking web pages
* Keep programs/files open scrolled to certain locations sometimes with things selected

In addition, three participants said that when interrupted they "rarely" or "very rarely" were able to resume the task they were working on prior to the interruption. Three of the participants said that they would not actively recommend their metawork strategies for other people, and two said that staying organized was "difficult".

Four participants were neutral to the idea of new tools to help them stay organized and three said that they would like to have such a tool/tools.

====IV. '''Discussion'''====
These results qunatiatively support our hypothesis that there is no clearly dominant set of metawork strategies employed by information workers. This highly fragemented landscape is surprising, even though most information workers work in a similar environment - at a desk, on the phone, in meetings - and with the same types of tools - computers, pens, paper, etc. We believe that this suggests that there are complex tradeoffs between these methods and that no single method is sufficient. We therefore believe that users will be better supported with a new set of software-based metawork tools.

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

=References=
<references/>

CS295J/Research proposal (draft 2)

2009-04-24T13:11:09Z

Eric Sodomka: /* Module Outputs */

CS295J/Research proposal (draft 2)

2009-04-24T13:10:55Z

Eric Sodomka: /* Module Outputs */

CS295J/Research proposal (draft 2)

2009-04-24T13:09:42Z

Eric Sodomka: /* Module Outputs */

CS295J/Research proposal (draft 2)

2009-04-24T13:09:09Z

Eric Sodomka: /* Aggregator Outputs */

CS295J/Research proposal (draft 2)

2009-04-24T12:59:28Z

Eric Sodomka: /* Module Outputs */

CS295J/Research proposal (draft 2)

2009-04-24T12:57:30Z

Eric Sodomka: /* Parallel Framework for Evaluation Modules */

CS295J/Research proposal (draft 2)

2009-04-24T12:56:51Z

Eric Sodomka: /* Parallel Framework for Evaluation Modules */

CS295J/Research proposal (draft 2)

2009-04-24T10:00:46Z

Eric Sodomka: /* Introduction */

CS295J

2009-04-21T21:49:22Z

Eric Sodomka:

'''Cognition, Human-Computer Interaction, and Visual Analysis''': in this graduate seminar we will learn about models of human cognition and perception and explore potential implications of the models on how computers and humans can interact effectively when performing scientific analyses. Participants will be responsible for reading assigned materials, taking turns guiding discussions of the readings, and preparing a final paper and presentation. It is recommended that participants have some background in at least one of the areas of study.

*Notes from Class
**[[/Class 1|Class 1]] 23 Jan 2009, [[/Theory of Visualization outline|Theory of Visualization outline]]
**[[/Class 2|Class 2]] 30 Jan 2009
**[[/Class 3|Class 3]] 6 Feb 2009
**[[/Class 4|Class 4]] 13 Feb 2009 (time changed: 10:30-12 and 1-2)
***[[/Posters from class 4|Posters from class 4]]
**[[/Class 5|Class 5]] 20 Feb 2009
**[[/Class 6|Class 6]] 27 Feb 2009, [[/Literature to read for class 6|Reading for class 6]], [[/Application Critiques|Application Critiques]], [[/Experiment results from class 6|Experiment results from class 6]]
**[[/Class 7|Class 7]] 6 Mar 2009, [[/Literature to read for class 7|Reading for class 7]], [[/Boards from class 7|Boards from class 7]] (preliminary results first sentence(s))
**[[/Class 8|Class 8]] 13 Mar 2009, [[/Literature to read for class 8|Reading for class 8]], [[/Proposal reviews from class 8|Proposal reviews from class 8]], [[/Boards from class 8|Boards from class 8]] (big picture intro stuff)
**[[/Class 9|Class 9]] 20 Mar 2009, [[/Literature to read for class 9|Reading for class 9]]
**Spring break
**[[/Class 10|Class 10]] 3 Apr 2009, [[/Literature to read for class 10|Reading for class 10]]
**[[/Class 11|Class 11]] 10 Apr 2009, [[/Literature to read for class 11|Reading for class 11]]
**[[/Class 12|Class 12]] 17 Apr 2009, [[/Literature to read for class 12|Reading for class 12]]
*[[/Assignments|Assignments]]
*'''[[/Research proposal|Research proposal]]'''
*'''[[CS295J/Research proposal (draft 2) | Research proposal (draft 2)]]'''
*[[/Relevant Journals and Conferences|Relevant Journals and Conferences]]
*[[/Literature|Literature]]
**[[/Literature week 2|Literature to read for class 3]] (2/6/09)
**[[/Literature to read for class 4|Reading for class 4]] (2/13/09)
**[[/Literature to read for class 5|Reading for class 5]] (2/20/09)

*[[/Rule Lists|Rule Lists]]
*[[/Model elements|Model elements]]
*[[/Project Concepts|Project Concepts]]

[[/How Tos|How Tos]] -- [[/Class Members' Pages|Class Members' Pages]] -- [http://groups.google.com/group/cs295j Mailing list archive]

[[Category:Courses]]

CS295J/Assignments

2009-04-21T16:59:36Z

Eric Sodomka: /* Assignment 12 (out April 21, due April 24, 2009) */

== Assignment 12 (out April 21, due April 24, 2009) ==

Complete all of your owned subsections for the [[CS295J/Research proposal (draft 2) | Research proposal (draft 2)]].

== Assignment 11 (out April 10, due April 17, 2009) ==

=== Part A (due Tuesday at noon) ===
First, revise any contribution that you added to the proposal to reflect our converging view of what we are proposing. Given the evolution of the proposal since the contributions were originally created, if you wish to start from a fresh set of contributions, you can outline them [[CS295J/Contributions for class 12 |here]].

Own one or more of the possible tasks below (enough to spend 8-10 hours on). Edit this page to indicate your ownership and to describe enough details of what you'll do for the group to avoid duplication. It's ok, for exmaple, for two folks to try using or extending CPM-GOMS with different extensions or a different application; indicating what your extension/application is, though would be helpful. It's also fine to add tasks to the list.

=== Part B (due class time) ===

Revise any parts of the preliminary results that you have added. That may include adding things that you would like to do, taking out pieces that don't fit into our worldview any longer, or revising things. The "CPM-GOMS for gmail" project presented today, for example, should go into the preliminary work section to establish viability of CPM-GOMS in the context of a more modern application.

Complete owned task(s).

=== Possible Tasks (also consider those from last week -- add here if you pick one) ===

* Possible additions to the proposed research
** Experiments with EEG to determine if they can inform modeling ('''Andrew Bragdon''' -- I will look into the EEG literature from the CHI community and determine what could be put in the proposal on this topic)

''CONCLUSION:'' Buying the BioSemi system used by Tan et. al. would cost about 50,000+ Euros. I think that we should add this to the budget, and incorporate EEG measurements into our methodology. Also, my conclusion is that making one giant equation which outputs only time may not be realistic. We should refine what our theory can predict, and we should consider working memory load as a model of cognitive load.

** Maybe similar experiments with muscle sensing or body motion
* Possible preliminary experiments
** Try some EEG
** Try some low-cost muscle tracking
** Try to decode some captured events
** Try to get some pupil-tracking thing going and sync'ed with other events (I found an open-source eye-tracking program called [http://www.inference.phy.cam.ac.uk/opengazer/ OpenGazer], but it's written for linux and might require considerable work to port. Would someone with a linux machine like to try it out? Adam)
** Try some light-weight JavaScript-based mouse tracking (will see how far I can get in 10 hours) ([[User:E J Kalafarski|E J Kalafarski]] 15:16, 14 April 2009 (UTC))
* Other
** Get "Human information processor model" concept into related work; ernestine is canonical example, but other more recent work too (Steven, Eric (Steven wrote core of the article, but I'll add to it as I come across other work))
** Revise proposal introduction again, emphasis on our new converging view ([[User:E J Kalafarski|E J Kalafarski]] 15:13, 14 April 2009 (UTC), [[User:Trevor|Trevor O'Brien]], Eric)
** Will revise and expand "big world" significance and contributions, based on the output/architecture we discussed last week ([[User:E J Kalafarski|E J Kalafarski]] 15:15, 14 April 2009 (UTC), [[User:Trevor|Trevor O'Brien]], Eric)
** Move 30-hour projects out of the contributions sections (Eric)
* CPM-GOMS
** Extend the cognition element in the CPM-GOMS model to account for dual-process theories of reasoning. (OWNER - Gideon; Jon)
** Extend the model by incorporating an 'Affect' output, instead of just relying on time as the only dependent variable. (SUGGESTED BY - Gideon; Ian (I like this idea))

== Assignment 10 (out April 3, due April 10, 2009) ==

=== Part A (due Tuesday at noon) ===
Own one or more of the possible tasks below (enough to spend 8-10 hours on). Edit this page to indicate your ownership and to describe enough details of what you'll do for the group to avoid duplication. It's ok, for exmaple, for two folks to try using or extending CPM-GOMS with different extensions or a different application; indicating what your extension/application is, though would be helpful. It's also fine to add tasks to the list.

=== Part B (due class time) ===

Complete owned task(s).

=== Possible Tasks ===

* read CPM-GOMS and consider as theme ('''[[User:E J Kalafarski|E J Kalafarski]]''', '''Eric''', '''Jon''', '''Gideon''', '''[[User:Trevor O'Brien|Trevor]]''', '''[[User:Steven Ellis|Steven]]''', '''Ian''')
* converge proposal (ID weaknesses & fix or propose fixes)
** make contributions and aims agree ('''[[User:Trevor O'Brien|Trevor]]''')
** make contributions and aims compelling and significant
*** intro acceptable (merge the two best) ('''[[User:E J Kalafarski|E J Kalafarski]]''': will rewrite intro, merging the two best "in progress" intros, with an emphasis on the project's interdisciplinary aspect)
*** improves world, lots of people, in big ways, long time ('''[[User:E J Kalafarski|E J Kalafarski]]''': will brainstorm and add to "big picture" contributions, but someone else should brainstorm this as well, '''Eric''')
*** extends knowledge, increased productivity (designers and users) ('''Eric''')
*** bg & significance section consistent with contribs/aims ('''Ian''')
* try CPM-GOMS, maybe w/ 1 extension ('''Gideon''', '''[[User:Steven Ellis|Steven]]''')
* what's input and output (architecture)? ('''[[User:E J Kalafarski|E J Kalafarski]]''': will attempt to come up with an architecture that accommodates all seven proposed modules and presents a simple, useful final product, ''Jon: Will brainstorm b/c I've been wondering about this for a long time'', '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')
* Gajos paper ('''[[User:E J Kalafarski|E J Kalafarski]]''': will read, consider architecture method, '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')

== Assignment 9 (out March 20, due April 3, 2009) ==

Part A is what we talked about in class -- a revision of the introduction to re-evaluate the big picture. Part B is the original assignment. While I've asked for both below, it may not be realistic given the time available. Please make sure to finish A by class time and to do as much of B as you can.

=== Part A, due noon Tuesday March 31 ===

Refine your introduction to be consistent with the theme of bringing knowledge of cognitive and perceptual modeling to bear in a principled way on human-computer interface design. Evaluate each of the suggested contributions for their impact, their relevance to the theme, and their cost. Triage as appropriate to achieve the best overall proposal given the 5 years with 5 people scope.

* [[CS295J/Proposal intros from class 9|Link to Intros]]
* [[CS295J/Triage for class 10|Link to Triages]]

=== Part B, due at class time ===

Complete your 30 hour preliminary work project and writeup in the proposal. It should explicitly support the contributions of the proposal. Consider this a hard, externally imposed deadline, i.e., you have to have something as finished as possible that could be reviewed by an outside reviewer. Trim scope, if necessary, but finish!

Pick another gap in the proposal and plug it. Any section that you have "owned" should also be "final" and reviewable.

====Gaps====
* <s>Significance (expand further with unified description of relatable components)</s>
* Specific contributions > Multi-modal HCI (question marks, needs content)
* Background > Workflow analysis > Interface improvements (incomplete)
* Research plan (we may have to cut this section, it is woefully underdeveloped)

Read [[CS295J/Literature to read for class 10|"must reads"]] -- add one if you want.

== Assignment 8 (out March 13, due March 20, 2009) ==
=== Part A, due Tuesday noon ===
* Outline a coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|here]]
* Select a gap to own
** '''Andrew Bragdon''': Metawork Support Tool proposal; will examine integrating this into the main proposal vs. making a separate proposal.
** '''EJ''': "Significance/intellectual merit" section is currently bare, I can take a stab at that. I believe this can/needs to incorporate a gap Trevor identified, "mapping between individual contributions and centralized theme of the proposal;" I'll try to start to illustrate the relationships between our individual projects. [[User:E J Kalafarski|E J Kalafarski]] 12:20, 17 March 2009 (UTC)
**'''Jon''': The "models of perception" section needs to be revised and expanded. I've changed "Gibsonianism" to "The Ecological Approach to Perception" and I've added a section on "The Computational Approach to Perception"; I will update these for Friday.
**'''Gideon''': More up-to-date background section on distributed cognition. I know Jon is doing this for the other areas, but I feel that this is very important.
** '''Trevor''': I'll take a pass through the ''Specific Aims'' section, which currently lacks specificity. --- [[User:Trevor O'Brien|Trevor O'Brien]] 15:29, 17 March 2009 (UTC)
**'''Eric''': I'll flesh out the ''Workflow Analysis'' section. What's there could be cleaned up, and more could be added, particularly on learning from interaction histories.
**'''Steven''': There's currently no background (slash anything) on the topic of embodied models of cognition, I'll fill in some background on that.
* suggest 0.5 "must read" [[CS295J/Literature to read for class 9|papers]]

=== Part B, due in class ===
* Finish writing the coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|still here]]
* Fill your gap
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results -- make sure to be consistent with your summary!
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 7 (out March 6, due March 13, 2009) ==
=== Part A, due Tuesday noon ===
* review [[CS295J/Research proposal|proposal]] for
** intellectual contribution (1-2 paragraphs)
** major gaps (bullet list) (don't duplicate gaps already listed)
** review goes [[CS295J/Proposal reviews from class 8|here]]
** feel free to improve any part of the proposal rather than criticizing it, if you wish :-)
* suggest 0.5 "must read" [[CS295J/Literature to read for class 8|papers]]

=== Part B, due in class ===
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 6 (out February 27, due March 6, 2009) ==

* If you are not familiar with NIH proposals, skim through [http://vis.cs.brown.edu/docs/pdf/bib/Laidlaw-2005-DA2.pdf this active proposal] to get a sense of the sections. The [http://vis.cs.brown.edu/docs/pdf/bib/NIH-2001-PHS.pdf guide to nih proposals] may also be helpful.
* Create some preliminary results
** Start the work you proposed in your poster for assignment 4
** Start the work someone else proposed :-)
** Create a more detailed critique-enabled workflow of a scientific user
** Working in pairs is fine; preferably, pair members should have different backgrounds
* Write them up as a subsection of the preliminary results section of the proposal.
** By Wednesday 9am have an outline of the section you will produce
*** This should be in past tense, as it will be when the work is done.
*** At the top level, it should state how the preliminary results enable the overall multi-year proposal or how they demonstrate feasibility of some questionable or risky part.
*** Check out the NIH proposal for examples, albeit in a different domain.
** By class have as much filled in from that outline as possible
* Add any "must reads" to the [[CS295J/Literature to read for class 6]] page. You are "expected" to add 1/2 of a reference.
* Read and be prepared to discuss the "must reads".
** Put in fictional placeholders for parts that are not done by Friday.
* By Wednesday 5pm e-mail constructive comments about at least 1/2 of the outlines to the entire class
* Success criteria for this assignment
*# Proposal section will demonstrate a feasibility or prerequisite for interesting research
*# Results by class are complete and concrete enough that they are interesting, even though some parts may be missing

== Assignment 5 (out February 20, due February 27, 2009) ==
# Videotape a 15-20 minute interactive session with a user doing a visually challenging task for which they are at least an advanced beginner (ie, they don't have to look up what to do, but they have not yet internalized the operations and made them subconscious). Analyze the session and report your observations and conclusions.

== Assignment 4 (out February 13, due February 20, 2009) ==

# Be prepared to decide on a subset of applications to critique. Flesh out at least one potential application in [[CS295J/Application Critiques]], including a specific workflow. Modify any others you want, particularly in terms of arguments for or against proceeding with them. These should be completed by Thursday noon.
#* Some possibilities: Google scholar, Mathematica, Tableau, Google notebook, Matlab, paper, media wiki, Ensight Gold, AVS, VisTrails. Note that a given piece of software could be represented more than once with a different workflow.
#* Possible criteria: main purpose is analysis, perhaps scientific; interative; scientific; amenable to cognition-driven improvemennts; interesting; fun
# Bring your ranking of the subset of (application+workflow)s we should critique
# Be prepared to decide on the [[CS295J/Model elements]] of cognition we will emulate to critique each application. Revise some part of this list so that it can be used as a concrete basis for evaluation.
# Add any "must reads" to the [[CS295J/Literature to read for class 5]] page. You are "expected" to add 1/3 of a reference.
# Read the "must reads".

== Assignment 3 (out February 6, due February 13) ==
* Flesh out and support a contribution within the proposal
** Add any new references to the literature section.
** Many of our readings date from 8+ years ago; check to make sure your contribution hasn't already been done.
** If any of the new references are "must reads", add to the [[CS295J/Literature to read for class 4]] (2/13/09) page. You are "expected" to add 1/2 of a reference.
** Estimate the impact of the contribution.
** Estimate the risk and costs of the contribution.
** Propose a 3-week (30 hour) result that you could create to demonstrate feasibility.
** Bring a printout of your contribution concept to class. It should be legible from 2 meters away, so use big text. You can hand-write it on posterboard or paste together printouts.
* Bring a list of holes in the proposal -- e.g., "background section needs something on workflow capture."

== Assignment 2 (out January 30, 2009) ==
* Refine literature short summaries to include the relationship to our project.
* Draft by tuesday noon a subsection in the background section of the [[CS295J/Research proposal]], making sure to include citations to the relevant materials in the literature page. Add any new references to the literature page.
* Read and comment/edit by Thursday noon all background sections
* Revise your background section by Friday noon (so David can print before class)

* Add to or refine one or more specific contribution in the [[CS295J/Research proposal]]; each contribution must have a list of ways it can be demonstrated. Some will become part of the preliminary results, others will be parts of the future work that will be proposed.
* Add to or refine one or more specific aim in the [[CS295J/Research proposal]]; make it consistent with the contribution you added.

* Identify the one additional most important paper for us to read this week also by Tuesday noon; be prepared to summarize relevance in 2 minutes in class
* Read those "most important" papers for class discussion.

* Be prepared to summarize to the class your contributions to the background, contributions, and aims sections.
* If there is preliminary work that will help to make decisions about contributions and aims, please get started on it (and be ready to report on what you'd like to do or what you have done).

== Assignment 1 (out January 23, 2009) ==
* spend 10 hours adding to any part of the wiki you think is relevant
* by Monday noon add any potential readings. If you've got a tentative summary evaluation, go ahead and add it. It's ok to edit folks summary evaluations, but try to make the result more accurate or precise without losing information.
* by Wednesday noon finish with any summary evaluation and also identify at least one as-relevant-as-possible reading as yours. Put your name on that entry in the reading list as the "owner" so that there are no duplicates.
* by Wednesday 5pm -- select 2 additional relevant readings that are owned and that you will read by Friday and be prepared to discuss. Put your name as a "discussant" in the reading list; there should be a max of two discussants per reading.
* by Friday class -- author a summary description, less than 250 words, in the wiki of how the reading you own relates to our project. Be prepared to describe, in two minutes, how your reading relates to the project. Also be prepared for everyone in class to discuss after your description. You may bring notes for yourself, but no slides. The wiki page for your reading will be displayed while you talk.
* by Friday class -- read and be prepared to discuss the other two readings you choose.
* by Friday class -- make one more wiki page titled "'''<Last-Name> week 1'''" with a list of the keys for the citations you added, the readings you summarized, the reading you presented, the two readings you were a discussant on, and any other readings you did in detail.
* Let me know if you have any kind of problems. You should be spending right around 10 hours -- if that's a problem, let's talk.
* The [[../How Tos|How Tos]] page has some tips. Edit or add as you find others.

CS295J/Research proposal (draft 2)

2009-04-21T16:55:45Z

Eric Sodomka: /* Specific Aims and Contributions (to be separated later) */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

= Specific Aims and Contributions (to be separated later) =
See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

In order to use this framework, a designer will have to provide:
* Functional specification - what are the possible interactions between the user and the application. This can be thought of as method signatures, with a name (e.g., setVolume), direction (to user or from user) and a list of value types (boolean, number, text, video, ...) for each interaction.
* GUI specification - a mapping of interactions to interface elements (e.g., setVolume is mapped to the grey knob in the bottom left corner with clockwise turning increasing the input number).
* Functional user traces - sequences of representative ways in which the application is used. Instead of writing them, the designer could have users use the application with a trial interface and then use our methods to generalize the user traces beyond the specific interface (The second method is depicted in the diagram). As a form of pre-processing, the system also generates an interaction transition matrix which lists the probability of each type of interaction given the previous interaction.
* Utility function - this is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

Each of the modules can use all of this information or a subset of it. Our approach stresses flexibility and the ability to give more meaningful feedback the more information is provided. After processing the information sent by the system of experts, the aggregator will output:
* An evaluation of the interface. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.
* Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''The user traces that are collected are tied to a specific interface. In order to use them with different interfaces to the same application, they should be generalized to be based only on the functional description of the application and the user's goal hierarchy. This would abstract away from actions like accessing a menu.''

''In addition to specific user traces, many modules could use a transition probability matrix based on interaction predictions.''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on perceived affordances and if possible a comparison to actual affordances.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

''Each person should come up with a single paragraph describing fictional (or not) preliminary results pertaining to their owned specific aims and contributions.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T16:33:53Z

Eric Sodomka: /* Outputs */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

= Specific Aims and Contributions (to be separated later) =
See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

== Inputs ==

== Outputs ==

Also passed as input is the utility function to optimize over. This utility function is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. For example, a user at NASA probably cares more about interface accuracy than speed. By passing this information to our committee of experts, we can create interfaces that are tuned to maximize the utility of a particular user type.

As output, the aggregator will provide an evaluation of the interface and a set of recommended improvements. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.

Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''The user traces that are collected are tied to a specific interface. In order to use them with different interfaces to the same application, they should be generalized to be based only on the functional description of the application and the user's goal hierarchy. This would abstract away from actions like accessing a menu.''

''In addition to specific user traces, many modules could use a transition probability matrix based on interaction predictions.''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on perceived affordances and if possible a comparison to actual affordances.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

''Each person should come up with a single paragraph describing fictional (or not) preliminary results pertaining to their owned specific aims and contributions.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T16:29:49Z

Eric Sodomka: /* Outputs */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

= Specific Aims and Contributions (to be separated later) =
See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

== Inputs ==

== Outputs ==

Also passed as input is the utility function to optimize over. This utility function is a weighting of various performance metrics (time, cognitive load, fatigue, etc.), where the weighting expresses the importance of a particular dimension to the user. The utility function is used by the aggregator to provide evaluations and recommendations appropriate to user preferences.

As output, the aggregator will provide an evaluation of the interface and a set of recommended improvements. Evaluations are expressed both in terms of the utility function components (i.e. time, fatigue, cognitive load, etc.), and in terms of the overall utility for this interface (as defined by the utility function). These evaluations are given in the form of an efficiency curve, where the utility received on each dimension can change as the user becomes more accustomed to the interface.

Suggested improvements for the GUI are also output. These suggestions are meant to optimize the utility function that was input to the system. If a user values accuracy over time, interface suggestions will be made accordingly.

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''The user traces that are collected are tied to a specific interface. In order to use them with different interfaces to the same application, they should be generalized to be based only on the functional description of the application and the user's goal hierarchy. This would abstract away from actions like accessing a menu.''

''In addition to specific user traces, many modules could use a transition probability matrix based on interaction predictions.''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on perceived affordances and if possible a comparison to actual affordances.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

''Each person should come up with a single paragraph describing fictional (or not) preliminary results pertaining to their owned specific aims and contributions.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T16:01:46Z

Eric Sodomka: /* Specific Aims and Contributions (to be separated later) */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

= Specific Aims and Contributions (to be separated later) =
See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

== Inputs ==

== Outputs ==

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''The user traces that are collected are tied to a specific interface. In order to use them with different interfaces to the same application, they should be generalized to be based only on the functional description of the application and the user's goal hierarchy. This would abstract away from actions like accessing a menu.''

''In addition to specific user traces, many modules could use a transition probability matrix based on interaction predictions.''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on perceived affordances and if possible a comparison to actual affordances.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

''Each person should come up with a single paragraph describing fictional (or not) preliminary results pertaining to their owned specific aims and contributions.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T16:01:14Z

Eric Sodomka: /* Specific Aims and Contributions (to be separated later) */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

Each person should add the background related to their specific aims.

* Steven Ellis - Cognitive models of HCI, including GOMS variations and ACT-R
* EJ - Design Guidelines
* Jon - Perception and Action
* Andrew - Multiple task environments
* Gideon - Cognition and dual systems
* Ian - Interface design process
* Trevor - User trace collection methods (especially any eye-tracking, EEG, ... you want to suggest using)

= Specific Aims and Contributions (to be separated later) =
See the [http://vrl.cs.brown.edu/wiki/images/b/b1/Flowchart2.pdf flowchart] for a visual overview of our aims.

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''The user traces that are collected are tied to a specific interface. In order to use them with different interfaces to the same application, they should be generalized to be based only on the functional description of the application and the user's goal hierarchy. This would abstract away from actions like accessing a menu.''

''In addition to specific user traces, many modules could use a transition probability matrix based on interaction predictions.''

== Parallel Framework for Evaluation Modules ==
* Owner: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

''This section will describe in more detail the inputs, outputs and architecture that were presented in the introduction.''

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on perceived affordances and if possible a comparison to actual affordances.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

== Integration into the Design Process ==
* Owner: [[User:Ian Spector | Ian Spector]]

''This section outlines the process of designing an HCI interface and at what stages our proposal fits in and how.''

= Preliminary Results =

''Each person should come up with a single paragraph describing fictional (or not) preliminary results pertaining to their owned specific aims and contributions.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

File:Flowchart2.pdf

2009-04-21T15:58:39Z

Eric Sodomka:

CS295J/Research proposal (draft 2)

2009-04-21T14:44:02Z

Eric Sodomka: /* Evaluation and Recommendation via Modules */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on perceived affordances and if possible a comparison to actual affordances.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

= Preliminary Results =
* Owner: [[User:Ian Spector | Ian Spector]]

''Initially, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T14:41:57Z

Eric Sodomka: /* Interruptions */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on perceived affordances and if possible a comparison to actual affordances.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

= Preliminary Results =
* Owner: [[User:Ian Spector | Ian Spector]]

''Initially, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T14:41:18Z

Eric Sodomka: /* Interaction Prediction */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This section could include an example or two of established design guidelines that could easily be implemented as modules.''

=== Fitts's Law ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Affordances ===
* Owner: [[User:Jon Ericson | Jon Ericson]]

''This simple module will provide interface evaluations and recommendations based on perceived affordances and if possible a comparison to actual affordances.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Working Memory Load ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''This module measures how much information the user needs to retain in memory while interacting with the interface and makes suggestions for improvements.''

=== Automaticity of Interaction ===
* Owner: [[User:Gideon Goldin | Gideon Goldin]]

''Measures how easily the interaction with the interface becomes automatic with experience and makes suggestions for improvements.''

= Preliminary Results =
* Owner: [[User:Ian Spector | Ian Spector]]

''Initially, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Assignments

2009-04-21T14:04:06Z

Eric Sodomka: /* Assignment 12 (out April 21, due April 24, 2009) */

== Assignment 12 (out April 21, due April 24, 2009) ==

Coming soon.

== Assignment 11 (out April 10, due April 17, 2009) ==

=== Part A (due Tuesday at noon) ===
First, revise any contribution that you added to the proposal to reflect our converging view of what we are proposing. Given the evolution of the proposal since the contributions were originally created, if you wish to start from a fresh set of contributions, you can outline them [[CS295J/Contributions for class 12 |here]].

Own one or more of the possible tasks below (enough to spend 8-10 hours on). Edit this page to indicate your ownership and to describe enough details of what you'll do for the group to avoid duplication. It's ok, for exmaple, for two folks to try using or extending CPM-GOMS with different extensions or a different application; indicating what your extension/application is, though would be helpful. It's also fine to add tasks to the list.

=== Part B (due class time) ===

Revise any parts of the preliminary results that you have added. That may include adding things that you would like to do, taking out pieces that don't fit into our worldview any longer, or revising things. The "CPM-GOMS for gmail" project presented today, for example, should go into the preliminary work section to establish viability of CPM-GOMS in the context of a more modern application.

Complete owned task(s).

=== Possible Tasks (also consider those from last week -- add here if you pick one) ===

* Possible additions to the proposed research
** Experiments with EEG to determine if they can inform modeling ('''Andrew Bragdon''' -- I will look into the EEG literature from the CHI community and determine what could be put in the proposal on this topic)

''CONCLUSION:'' Buying the BioSemi system used by Tan et. al. would cost about 50,000+ Euros. I think that we should add this to the budget, and incorporate EEG measurements into our methodology. Also, my conclusion is that making one giant equation which outputs only time may not be realistic. We should refine what our theory can predict, and we should consider working memory load as a model of cognitive load.

** Maybe similar experiments with muscle sensing or body motion
* Possible preliminary experiments
** Try some EEG
** Try some low-cost muscle tracking
** Try to decode some captured events
** Try to get some pupil-tracking thing going and sync'ed with other events (I found an open-source eye-tracking program called [http://www.inference.phy.cam.ac.uk/opengazer/ OpenGazer], but it's written for linux and might require considerable work to port. Would someone with a linux machine like to try it out? Adam)
** Try some light-weight JavaScript-based mouse tracking (will see how far I can get in 10 hours) ([[User:E J Kalafarski|E J Kalafarski]] 15:16, 14 April 2009 (UTC))
* Other
** Get "Human information processor model" concept into related work; ernestine is canonical example, but other more recent work too (Steven, Eric (Steven wrote core of the article, but I'll add to it as I come across other work))
** Revise proposal introduction again, emphasis on our new converging view ([[User:E J Kalafarski|E J Kalafarski]] 15:13, 14 April 2009 (UTC), [[User:Trevor|Trevor O'Brien]], Eric)
** Will revise and expand "big world" significance and contributions, based on the output/architecture we discussed last week ([[User:E J Kalafarski|E J Kalafarski]] 15:15, 14 April 2009 (UTC), [[User:Trevor|Trevor O'Brien]], Eric)
** Move 30-hour projects out of the contributions sections (Eric)
* CPM-GOMS
** Extend the cognition element in the CPM-GOMS model to account for dual-process theories of reasoning. (OWNER - Gideon; Jon)
** Extend the model by incorporating an 'Affect' output, instead of just relying on time as the only dependent variable. (SUGGESTED BY - Gideon; Ian (I like this idea))

== Assignment 10 (out April 3, due April 10, 2009) ==

=== Part A (due Tuesday at noon) ===
Own one or more of the possible tasks below (enough to spend 8-10 hours on). Edit this page to indicate your ownership and to describe enough details of what you'll do for the group to avoid duplication. It's ok, for exmaple, for two folks to try using or extending CPM-GOMS with different extensions or a different application; indicating what your extension/application is, though would be helpful. It's also fine to add tasks to the list.

=== Part B (due class time) ===

Complete owned task(s).

=== Possible Tasks ===

* read CPM-GOMS and consider as theme ('''[[User:E J Kalafarski|E J Kalafarski]]''', '''Eric''', '''Jon''', '''Gideon''', '''[[User:Trevor O'Brien|Trevor]]''', '''[[User:Steven Ellis|Steven]]''', '''Ian''')
* converge proposal (ID weaknesses & fix or propose fixes)
** make contributions and aims agree ('''[[User:Trevor O'Brien|Trevor]]''')
** make contributions and aims compelling and significant
*** intro acceptable (merge the two best) ('''[[User:E J Kalafarski|E J Kalafarski]]''': will rewrite intro, merging the two best "in progress" intros, with an emphasis on the project's interdisciplinary aspect)
*** improves world, lots of people, in big ways, long time ('''[[User:E J Kalafarski|E J Kalafarski]]''': will brainstorm and add to "big picture" contributions, but someone else should brainstorm this as well, '''Eric''')
*** extends knowledge, increased productivity (designers and users) ('''Eric''')
*** bg & significance section consistent with contribs/aims ('''Ian''')
* try CPM-GOMS, maybe w/ 1 extension ('''Gideon''', '''[[User:Steven Ellis|Steven]]''')
* what's input and output (architecture)? ('''[[User:E J Kalafarski|E J Kalafarski]]''': will attempt to come up with an architecture that accommodates all seven proposed modules and presents a simple, useful final product, ''Jon: Will brainstorm b/c I've been wondering about this for a long time'', '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')
* Gajos paper ('''[[User:E J Kalafarski|E J Kalafarski]]''': will read, consider architecture method, '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')

== Assignment 9 (out March 20, due April 3, 2009) ==

Part A is what we talked about in class -- a revision of the introduction to re-evaluate the big picture. Part B is the original assignment. While I've asked for both below, it may not be realistic given the time available. Please make sure to finish A by class time and to do as much of B as you can.

=== Part A, due noon Tuesday March 31 ===

Refine your introduction to be consistent with the theme of bringing knowledge of cognitive and perceptual modeling to bear in a principled way on human-computer interface design. Evaluate each of the suggested contributions for their impact, their relevance to the theme, and their cost. Triage as appropriate to achieve the best overall proposal given the 5 years with 5 people scope.

* [[CS295J/Proposal intros from class 9|Link to Intros]]
* [[CS295J/Triage for class 10|Link to Triages]]

=== Part B, due at class time ===

Complete your 30 hour preliminary work project and writeup in the proposal. It should explicitly support the contributions of the proposal. Consider this a hard, externally imposed deadline, i.e., you have to have something as finished as possible that could be reviewed by an outside reviewer. Trim scope, if necessary, but finish!

Pick another gap in the proposal and plug it. Any section that you have "owned" should also be "final" and reviewable.

====Gaps====
* <s>Significance (expand further with unified description of relatable components)</s>
* Specific contributions > Multi-modal HCI (question marks, needs content)
* Background > Workflow analysis > Interface improvements (incomplete)
* Research plan (we may have to cut this section, it is woefully underdeveloped)

Read [[CS295J/Literature to read for class 10|"must reads"]] -- add one if you want.

== Assignment 8 (out March 13, due March 20, 2009) ==
=== Part A, due Tuesday noon ===
* Outline a coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|here]]
* Select a gap to own
** '''Andrew Bragdon''': Metawork Support Tool proposal; will examine integrating this into the main proposal vs. making a separate proposal.
** '''EJ''': "Significance/intellectual merit" section is currently bare, I can take a stab at that. I believe this can/needs to incorporate a gap Trevor identified, "mapping between individual contributions and centralized theme of the proposal;" I'll try to start to illustrate the relationships between our individual projects. [[User:E J Kalafarski|E J Kalafarski]] 12:20, 17 March 2009 (UTC)
**'''Jon''': The "models of perception" section needs to be revised and expanded. I've changed "Gibsonianism" to "The Ecological Approach to Perception" and I've added a section on "The Computational Approach to Perception"; I will update these for Friday.
**'''Gideon''': More up-to-date background section on distributed cognition. I know Jon is doing this for the other areas, but I feel that this is very important.
** '''Trevor''': I'll take a pass through the ''Specific Aims'' section, which currently lacks specificity. --- [[User:Trevor O'Brien|Trevor O'Brien]] 15:29, 17 March 2009 (UTC)
**'''Eric''': I'll flesh out the ''Workflow Analysis'' section. What's there could be cleaned up, and more could be added, particularly on learning from interaction histories.
**'''Steven''': There's currently no background (slash anything) on the topic of embodied models of cognition, I'll fill in some background on that.
* suggest 0.5 "must read" [[CS295J/Literature to read for class 9|papers]]

=== Part B, due in class ===
* Finish writing the coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|still here]]
* Fill your gap
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results -- make sure to be consistent with your summary!
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 7 (out March 6, due March 13, 2009) ==
=== Part A, due Tuesday noon ===
* review [[CS295J/Research proposal|proposal]] for
** intellectual contribution (1-2 paragraphs)
** major gaps (bullet list) (don't duplicate gaps already listed)
** review goes [[CS295J/Proposal reviews from class 8|here]]
** feel free to improve any part of the proposal rather than criticizing it, if you wish :-)
* suggest 0.5 "must read" [[CS295J/Literature to read for class 8|papers]]

=== Part B, due in class ===
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 6 (out February 27, due March 6, 2009) ==

* If you are not familiar with NIH proposals, skim through [http://vis.cs.brown.edu/docs/pdf/bib/Laidlaw-2005-DA2.pdf this active proposal] to get a sense of the sections. The [http://vis.cs.brown.edu/docs/pdf/bib/NIH-2001-PHS.pdf guide to nih proposals] may also be helpful.
* Create some preliminary results
** Start the work you proposed in your poster for assignment 4
** Start the work someone else proposed :-)
** Create a more detailed critique-enabled workflow of a scientific user
** Working in pairs is fine; preferably, pair members should have different backgrounds
* Write them up as a subsection of the preliminary results section of the proposal.
** By Wednesday 9am have an outline of the section you will produce
*** This should be in past tense, as it will be when the work is done.
*** At the top level, it should state how the preliminary results enable the overall multi-year proposal or how they demonstrate feasibility of some questionable or risky part.
*** Check out the NIH proposal for examples, albeit in a different domain.
** By class have as much filled in from that outline as possible
* Add any "must reads" to the [[CS295J/Literature to read for class 6]] page. You are "expected" to add 1/2 of a reference.
* Read and be prepared to discuss the "must reads".
** Put in fictional placeholders for parts that are not done by Friday.
* By Wednesday 5pm e-mail constructive comments about at least 1/2 of the outlines to the entire class
* Success criteria for this assignment
*# Proposal section will demonstrate a feasibility or prerequisite for interesting research
*# Results by class are complete and concrete enough that they are interesting, even though some parts may be missing

== Assignment 5 (out February 20, due February 27, 2009) ==
# Videotape a 15-20 minute interactive session with a user doing a visually challenging task for which they are at least an advanced beginner (ie, they don't have to look up what to do, but they have not yet internalized the operations and made them subconscious). Analyze the session and report your observations and conclusions.

== Assignment 4 (out February 13, due February 20, 2009) ==

# Be prepared to decide on a subset of applications to critique. Flesh out at least one potential application in [[CS295J/Application Critiques]], including a specific workflow. Modify any others you want, particularly in terms of arguments for or against proceeding with them. These should be completed by Thursday noon.
#* Some possibilities: Google scholar, Mathematica, Tableau, Google notebook, Matlab, paper, media wiki, Ensight Gold, AVS, VisTrails. Note that a given piece of software could be represented more than once with a different workflow.
#* Possible criteria: main purpose is analysis, perhaps scientific; interative; scientific; amenable to cognition-driven improvemennts; interesting; fun
# Bring your ranking of the subset of (application+workflow)s we should critique
# Be prepared to decide on the [[CS295J/Model elements]] of cognition we will emulate to critique each application. Revise some part of this list so that it can be used as a concrete basis for evaluation.
# Add any "must reads" to the [[CS295J/Literature to read for class 5]] page. You are "expected" to add 1/3 of a reference.
# Read the "must reads".

== Assignment 3 (out February 6, due February 13) ==
* Flesh out and support a contribution within the proposal
** Add any new references to the literature section.
** Many of our readings date from 8+ years ago; check to make sure your contribution hasn't already been done.
** If any of the new references are "must reads", add to the [[CS295J/Literature to read for class 4]] (2/13/09) page. You are "expected" to add 1/2 of a reference.
** Estimate the impact of the contribution.
** Estimate the risk and costs of the contribution.
** Propose a 3-week (30 hour) result that you could create to demonstrate feasibility.
** Bring a printout of your contribution concept to class. It should be legible from 2 meters away, so use big text. You can hand-write it on posterboard or paste together printouts.
* Bring a list of holes in the proposal -- e.g., "background section needs something on workflow capture."

== Assignment 2 (out January 30, 2009) ==
* Refine literature short summaries to include the relationship to our project.
* Draft by tuesday noon a subsection in the background section of the [[CS295J/Research proposal]], making sure to include citations to the relevant materials in the literature page. Add any new references to the literature page.
* Read and comment/edit by Thursday noon all background sections
* Revise your background section by Friday noon (so David can print before class)

* Add to or refine one or more specific contribution in the [[CS295J/Research proposal]]; each contribution must have a list of ways it can be demonstrated. Some will become part of the preliminary results, others will be parts of the future work that will be proposed.
* Add to or refine one or more specific aim in the [[CS295J/Research proposal]]; make it consistent with the contribution you added.

* Identify the one additional most important paper for us to read this week also by Tuesday noon; be prepared to summarize relevance in 2 minutes in class
* Read those "most important" papers for class discussion.

* Be prepared to summarize to the class your contributions to the background, contributions, and aims sections.
* If there is preliminary work that will help to make decisions about contributions and aims, please get started on it (and be ready to report on what you'd like to do or what you have done).

== Assignment 1 (out January 23, 2009) ==
* spend 10 hours adding to any part of the wiki you think is relevant
* by Monday noon add any potential readings. If you've got a tentative summary evaluation, go ahead and add it. It's ok to edit folks summary evaluations, but try to make the result more accurate or precise without losing information.
* by Wednesday noon finish with any summary evaluation and also identify at least one as-relevant-as-possible reading as yours. Put your name on that entry in the reading list as the "owner" so that there are no duplicates.
* by Wednesday 5pm -- select 2 additional relevant readings that are owned and that you will read by Friday and be prepared to discuss. Put your name as a "discussant" in the reading list; there should be a max of two discussants per reading.
* by Friday class -- author a summary description, less than 250 words, in the wiki of how the reading you own relates to our project. Be prepared to describe, in two minutes, how your reading relates to the project. Also be prepared for everyone in class to discuss after your description. You may bring notes for yourself, but no slides. The wiki page for your reading will be displayed while you talk.
* by Friday class -- read and be prepared to discuss the other two readings you choose.
* by Friday class -- make one more wiki page titled "'''<Last-Name> week 1'''" with a list of the keys for the citations you added, the readings you summarized, the reading you presented, the two readings you were a discussant on, and any other readings you did in detail.
* Let me know if you have any kind of problems. You should be spending right around 10 hours -- if that's a problem, let's talk.
* The [[../How Tos|How Tos]] page has some tips. Edit or add as you find others.

CS295J/Assignments

2009-04-21T14:03:55Z

Eric Sodomka:

== Assignment 12 (out April 21, due April 24, 2009) ==

== Assignment 11 (out April 10, due April 17, 2009) ==

=== Part A (due Tuesday at noon) ===
First, revise any contribution that you added to the proposal to reflect our converging view of what we are proposing. Given the evolution of the proposal since the contributions were originally created, if you wish to start from a fresh set of contributions, you can outline them [[CS295J/Contributions for class 12 |here]].

Own one or more of the possible tasks below (enough to spend 8-10 hours on). Edit this page to indicate your ownership and to describe enough details of what you'll do for the group to avoid duplication. It's ok, for exmaple, for two folks to try using or extending CPM-GOMS with different extensions or a different application; indicating what your extension/application is, though would be helpful. It's also fine to add tasks to the list.

=== Part B (due class time) ===

Revise any parts of the preliminary results that you have added. That may include adding things that you would like to do, taking out pieces that don't fit into our worldview any longer, or revising things. The "CPM-GOMS for gmail" project presented today, for example, should go into the preliminary work section to establish viability of CPM-GOMS in the context of a more modern application.

Complete owned task(s).

=== Possible Tasks (also consider those from last week -- add here if you pick one) ===

* Possible additions to the proposed research
** Experiments with EEG to determine if they can inform modeling ('''Andrew Bragdon''' -- I will look into the EEG literature from the CHI community and determine what could be put in the proposal on this topic)

''CONCLUSION:'' Buying the BioSemi system used by Tan et. al. would cost about 50,000+ Euros. I think that we should add this to the budget, and incorporate EEG measurements into our methodology. Also, my conclusion is that making one giant equation which outputs only time may not be realistic. We should refine what our theory can predict, and we should consider working memory load as a model of cognitive load.

** Maybe similar experiments with muscle sensing or body motion
* Possible preliminary experiments
** Try some EEG
** Try some low-cost muscle tracking
** Try to decode some captured events
** Try to get some pupil-tracking thing going and sync'ed with other events (I found an open-source eye-tracking program called [http://www.inference.phy.cam.ac.uk/opengazer/ OpenGazer], but it's written for linux and might require considerable work to port. Would someone with a linux machine like to try it out? Adam)
** Try some light-weight JavaScript-based mouse tracking (will see how far I can get in 10 hours) ([[User:E J Kalafarski|E J Kalafarski]] 15:16, 14 April 2009 (UTC))
* Other
** Get "Human information processor model" concept into related work; ernestine is canonical example, but other more recent work too (Steven, Eric (Steven wrote core of the article, but I'll add to it as I come across other work))
** Revise proposal introduction again, emphasis on our new converging view ([[User:E J Kalafarski|E J Kalafarski]] 15:13, 14 April 2009 (UTC), [[User:Trevor|Trevor O'Brien]], Eric)
** Will revise and expand "big world" significance and contributions, based on the output/architecture we discussed last week ([[User:E J Kalafarski|E J Kalafarski]] 15:15, 14 April 2009 (UTC), [[User:Trevor|Trevor O'Brien]], Eric)
** Move 30-hour projects out of the contributions sections (Eric)
* CPM-GOMS
** Extend the cognition element in the CPM-GOMS model to account for dual-process theories of reasoning. (OWNER - Gideon; Jon)
** Extend the model by incorporating an 'Affect' output, instead of just relying on time as the only dependent variable. (SUGGESTED BY - Gideon; Ian (I like this idea))

== Assignment 10 (out April 3, due April 10, 2009) ==

=== Part A (due Tuesday at noon) ===
Own one or more of the possible tasks below (enough to spend 8-10 hours on). Edit this page to indicate your ownership and to describe enough details of what you'll do for the group to avoid duplication. It's ok, for exmaple, for two folks to try using or extending CPM-GOMS with different extensions or a different application; indicating what your extension/application is, though would be helpful. It's also fine to add tasks to the list.

=== Part B (due class time) ===

Complete owned task(s).

=== Possible Tasks ===

* read CPM-GOMS and consider as theme ('''[[User:E J Kalafarski|E J Kalafarski]]''', '''Eric''', '''Jon''', '''Gideon''', '''[[User:Trevor O'Brien|Trevor]]''', '''[[User:Steven Ellis|Steven]]''', '''Ian''')
* converge proposal (ID weaknesses & fix or propose fixes)
** make contributions and aims agree ('''[[User:Trevor O'Brien|Trevor]]''')
** make contributions and aims compelling and significant
*** intro acceptable (merge the two best) ('''[[User:E J Kalafarski|E J Kalafarski]]''': will rewrite intro, merging the two best "in progress" intros, with an emphasis on the project's interdisciplinary aspect)
*** improves world, lots of people, in big ways, long time ('''[[User:E J Kalafarski|E J Kalafarski]]''': will brainstorm and add to "big picture" contributions, but someone else should brainstorm this as well, '''Eric''')
*** extends knowledge, increased productivity (designers and users) ('''Eric''')
*** bg & significance section consistent with contribs/aims ('''Ian''')
* try CPM-GOMS, maybe w/ 1 extension ('''Gideon''', '''[[User:Steven Ellis|Steven]]''')
* what's input and output (architecture)? ('''[[User:E J Kalafarski|E J Kalafarski]]''': will attempt to come up with an architecture that accommodates all seven proposed modules and presents a simple, useful final product, ''Jon: Will brainstorm b/c I've been wondering about this for a long time'', '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')
* Gajos paper ('''[[User:E J Kalafarski|E J Kalafarski]]''': will read, consider architecture method, '''Adam''', '''[[User:Trevor O'Brien|Trevor]]''')

== Assignment 9 (out March 20, due April 3, 2009) ==

Part A is what we talked about in class -- a revision of the introduction to re-evaluate the big picture. Part B is the original assignment. While I've asked for both below, it may not be realistic given the time available. Please make sure to finish A by class time and to do as much of B as you can.

=== Part A, due noon Tuesday March 31 ===

Refine your introduction to be consistent with the theme of bringing knowledge of cognitive and perceptual modeling to bear in a principled way on human-computer interface design. Evaluate each of the suggested contributions for their impact, their relevance to the theme, and their cost. Triage as appropriate to achieve the best overall proposal given the 5 years with 5 people scope.

* [[CS295J/Proposal intros from class 9|Link to Intros]]
* [[CS295J/Triage for class 10|Link to Triages]]

=== Part B, due at class time ===

Complete your 30 hour preliminary work project and writeup in the proposal. It should explicitly support the contributions of the proposal. Consider this a hard, externally imposed deadline, i.e., you have to have something as finished as possible that could be reviewed by an outside reviewer. Trim scope, if necessary, but finish!

Pick another gap in the proposal and plug it. Any section that you have "owned" should also be "final" and reviewable.

====Gaps====
* <s>Significance (expand further with unified description of relatable components)</s>
* Specific contributions > Multi-modal HCI (question marks, needs content)
* Background > Workflow analysis > Interface improvements (incomplete)
* Research plan (we may have to cut this section, it is woefully underdeveloped)

Read [[CS295J/Literature to read for class 10|"must reads"]] -- add one if you want.

== Assignment 8 (out March 13, due March 20, 2009) ==
=== Part A, due Tuesday noon ===
* Outline a coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|here]]
* Select a gap to own
** '''Andrew Bragdon''': Metawork Support Tool proposal; will examine integrating this into the main proposal vs. making a separate proposal.
** '''EJ''': "Significance/intellectual merit" section is currently bare, I can take a stab at that. I believe this can/needs to incorporate a gap Trevor identified, "mapping between individual contributions and centralized theme of the proposal;" I'll try to start to illustrate the relationships between our individual projects. [[User:E J Kalafarski|E J Kalafarski]] 12:20, 17 March 2009 (UTC)
**'''Jon''': The "models of perception" section needs to be revised and expanded. I've changed "Gibsonianism" to "The Ecological Approach to Perception" and I've added a section on "The Computational Approach to Perception"; I will update these for Friday.
**'''Gideon''': More up-to-date background section on distributed cognition. I know Jon is doing this for the other areas, but I feel that this is very important.
** '''Trevor''': I'll take a pass through the ''Specific Aims'' section, which currently lacks specificity. --- [[User:Trevor O'Brien|Trevor O'Brien]] 15:29, 17 March 2009 (UTC)
**'''Eric''': I'll flesh out the ''Workflow Analysis'' section. What's there could be cleaned up, and more could be added, particularly on learning from interaction histories.
**'''Steven''': There's currently no background (slash anything) on the topic of embodied models of cognition, I'll fill in some background on that.
* suggest 0.5 "must read" [[CS295J/Literature to read for class 9|papers]]

=== Part B, due in class ===
* Finish writing the coherent 250 word summary to a coherent proposal [[CS295J/Proposal intros from class 9|still here]]
* Fill your gap
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results -- make sure to be consistent with your summary!
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 7 (out March 6, due March 13, 2009) ==
=== Part A, due Tuesday noon ===
* review [[CS295J/Research proposal|proposal]] for
** intellectual contribution (1-2 paragraphs)
** major gaps (bullet list) (don't duplicate gaps already listed)
** review goes [[CS295J/Proposal reviews from class 8|here]]
** feel free to improve any part of the proposal rather than criticizing it, if you wish :-)
* suggest 0.5 "must read" [[CS295J/Literature to read for class 8|papers]]

=== Part B, due in class ===
* read "must read"s for discussion w.r.t. proposal relevance
* get to another week's worth of results in your preliminary results
* be prepared to tell us about those new results in class, tied to the intellectual claims

== Assignment 6 (out February 27, due March 6, 2009) ==

* If you are not familiar with NIH proposals, skim through [http://vis.cs.brown.edu/docs/pdf/bib/Laidlaw-2005-DA2.pdf this active proposal] to get a sense of the sections. The [http://vis.cs.brown.edu/docs/pdf/bib/NIH-2001-PHS.pdf guide to nih proposals] may also be helpful.
* Create some preliminary results
** Start the work you proposed in your poster for assignment 4
** Start the work someone else proposed :-)
** Create a more detailed critique-enabled workflow of a scientific user
** Working in pairs is fine; preferably, pair members should have different backgrounds
* Write them up as a subsection of the preliminary results section of the proposal.
** By Wednesday 9am have an outline of the section you will produce
*** This should be in past tense, as it will be when the work is done.
*** At the top level, it should state how the preliminary results enable the overall multi-year proposal or how they demonstrate feasibility of some questionable or risky part.
*** Check out the NIH proposal for examples, albeit in a different domain.
** By class have as much filled in from that outline as possible
* Add any "must reads" to the [[CS295J/Literature to read for class 6]] page. You are "expected" to add 1/2 of a reference.
* Read and be prepared to discuss the "must reads".
** Put in fictional placeholders for parts that are not done by Friday.
* By Wednesday 5pm e-mail constructive comments about at least 1/2 of the outlines to the entire class
* Success criteria for this assignment
*# Proposal section will demonstrate a feasibility or prerequisite for interesting research
*# Results by class are complete and concrete enough that they are interesting, even though some parts may be missing

== Assignment 5 (out February 20, due February 27, 2009) ==
# Videotape a 15-20 minute interactive session with a user doing a visually challenging task for which they are at least an advanced beginner (ie, they don't have to look up what to do, but they have not yet internalized the operations and made them subconscious). Analyze the session and report your observations and conclusions.

== Assignment 4 (out February 13, due February 20, 2009) ==

# Be prepared to decide on a subset of applications to critique. Flesh out at least one potential application in [[CS295J/Application Critiques]], including a specific workflow. Modify any others you want, particularly in terms of arguments for or against proceeding with them. These should be completed by Thursday noon.
#* Some possibilities: Google scholar, Mathematica, Tableau, Google notebook, Matlab, paper, media wiki, Ensight Gold, AVS, VisTrails. Note that a given piece of software could be represented more than once with a different workflow.
#* Possible criteria: main purpose is analysis, perhaps scientific; interative; scientific; amenable to cognition-driven improvemennts; interesting; fun
# Bring your ranking of the subset of (application+workflow)s we should critique
# Be prepared to decide on the [[CS295J/Model elements]] of cognition we will emulate to critique each application. Revise some part of this list so that it can be used as a concrete basis for evaluation.
# Add any "must reads" to the [[CS295J/Literature to read for class 5]] page. You are "expected" to add 1/3 of a reference.
# Read the "must reads".

== Assignment 3 (out February 6, due February 13) ==
* Flesh out and support a contribution within the proposal
** Add any new references to the literature section.
** Many of our readings date from 8+ years ago; check to make sure your contribution hasn't already been done.
** If any of the new references are "must reads", add to the [[CS295J/Literature to read for class 4]] (2/13/09) page. You are "expected" to add 1/2 of a reference.
** Estimate the impact of the contribution.
** Estimate the risk and costs of the contribution.
** Propose a 3-week (30 hour) result that you could create to demonstrate feasibility.
** Bring a printout of your contribution concept to class. It should be legible from 2 meters away, so use big text. You can hand-write it on posterboard or paste together printouts.
* Bring a list of holes in the proposal -- e.g., "background section needs something on workflow capture."

== Assignment 2 (out January 30, 2009) ==
* Refine literature short summaries to include the relationship to our project.
* Draft by tuesday noon a subsection in the background section of the [[CS295J/Research proposal]], making sure to include citations to the relevant materials in the literature page. Add any new references to the literature page.
* Read and comment/edit by Thursday noon all background sections
* Revise your background section by Friday noon (so David can print before class)

* Add to or refine one or more specific contribution in the [[CS295J/Research proposal]]; each contribution must have a list of ways it can be demonstrated. Some will become part of the preliminary results, others will be parts of the future work that will be proposed.
* Add to or refine one or more specific aim in the [[CS295J/Research proposal]]; make it consistent with the contribution you added.

* Identify the one additional most important paper for us to read this week also by Tuesday noon; be prepared to summarize relevance in 2 minutes in class
* Read those "most important" papers for class discussion.

* Be prepared to summarize to the class your contributions to the background, contributions, and aims sections.
* If there is preliminary work that will help to make decisions about contributions and aims, please get started on it (and be ready to report on what you'd like to do or what you have done).

== Assignment 1 (out January 23, 2009) ==
* spend 10 hours adding to any part of the wiki you think is relevant
* by Monday noon add any potential readings. If you've got a tentative summary evaluation, go ahead and add it. It's ok to edit folks summary evaluations, but try to make the result more accurate or precise without losing information.
* by Wednesday noon finish with any summary evaluation and also identify at least one as-relevant-as-possible reading as yours. Put your name on that entry in the reading list as the "owner" so that there are no duplicates.
* by Wednesday 5pm -- select 2 additional relevant readings that are owned and that you will read by Friday and be prepared to discuss. Put your name as a "discussant" in the reading list; there should be a max of two discussants per reading.
* by Friday class -- author a summary description, less than 250 words, in the wiki of how the reading you own relates to our project. Be prepared to describe, in two minutes, how your reading relates to the project. Also be prepared for everyone in class to discuss after your description. You may bring notes for yourself, but no slides. The wiki page for your reading will be displayed while you talk.
* by Friday class -- read and be prepared to discuss the other two readings you choose.
* by Friday class -- make one more wiki page titled "'''<Last-Name> week 1'''" with a list of the keys for the citations you added, the readings you summarized, the reading you presented, the two readings you were a discussant on, and any other readings you did in detail.
* Let me know if you have any kind of problems. You should be spending right around 10 hours -- if that's a problem, let's talk.
* The [[../How Tos|How Tos]] page has some tips. Edit or add as you find others.

Talk:CS295J/Research proposal (draft 2)

2009-04-21T11:22:13Z

Eric Sodomka:

People in the class to assign tasks:
* [[User:Adam Darlow | Adam Darlow]]
* [[User:Eric Sodomka | Eric Sodomka]]
* [[User:E J Kalafarski | E J Kalafarski]]
* [[User:Trevor O'Brien | Trevor O'Brien]]
* [[User:Ian Spector | Ian Spector]]
* [[User:Andrew Bragdon | Andrew Bragdon]]
* [[User:Steven Ellis | Steven Ellis]]
* [[User:Gideon Goldin | Gideon Goldin]]
* [[User:Jon Ericson | Jon Ericson]]

TODO:
* More evenly distribute the work amongst people in the class
* Clean up the intro
* Add an intro to the methodology section
* Breakdown the related work section into subsections?
* Add description to 'Generalizing User Traces' section
* Add a figure?

Talk:CS295J/Research proposal (draft 2)

2009-04-21T11:21:42Z

Eric Sodomka:

CS295J/Research proposal (draft 2)

2009-04-21T11:15:09Z

Eric Sodomka: /* Cognitive/HCI Guidelines */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]], [[User:Jon Ericson | Jon Ericson]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Gideon Goldin | Gideon Goldin]], [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== Cognitive/HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This module weights various interface guidelines from the HCI and cognitive science literature, and uses these weights to provide suggested improvements for the given interface. It identifies interface elements that are detrimental to user performance and suggests effective alternatives.''

=== Fitts's Law ===

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Interaction Prediction ===
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given the history of user interactions, we may be able to improve user performance by modifying the interface so that frequently performed interactions from a given state are more readily accessible to the user. This module recommendations such modifications based on user traces.''

= Preliminary Results =
* Owner: [[User:Ian Spector | Ian Spector]]

''Initially, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T11:14:39Z

Eric Sodomka: /* Cognitive/HCI Guidelines */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]], [[User:Jon Ericson | Jon Ericson]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Gideon Goldin | Gideon Goldin]], [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== Cognitive/HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

''This module weights various interface guidelines from HCI and cognitive science, and uses these weights to provide suggested improvements for the given interface. It identifies interface elements that are detrimental to user performance and suggests effective alternatives.''

=== Fitts's Law ===

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Interaction Prediction ===
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given the history of user interactions, we may be able to improve user performance by modifying the interface so that frequently performed interactions from a given state are more readily accessible to the user. This module recommendations such modifications based on user traces.''

= Preliminary Results =
* Owner: [[User:Ian Spector | Ian Spector]]

''Initially, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T11:07:01Z

Eric Sodomka: /* Interaction Prediction */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]], [[User:Jon Ericson | Jon Ericson]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Gideon Goldin | Gideon Goldin]], [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== Cognitive/HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

=== Fitts's Law ===

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Interaction Prediction ===
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given the history of user interactions, we may be able to improve user performance by modifying the interface so that frequently performed interactions from a given state are more readily accessible to the user. This module recommendations such modifications based on user traces.''

= Preliminary Results =
* Owner: [[User:Ian Spector | Ian Spector]]

''Initially, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T11:04:57Z

Eric Sodomka: /* Interaction Prediction */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]], [[User:Jon Ericson | Jon Ericson]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Gideon Goldin | Gideon Goldin]], [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== Cognitive/HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

=== Fitts's Law ===

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Interaction Prediction ===
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given the history of user interactions, we may be able to improve user performance by modifying the interface so that frequently performed interactions from a given state are more readily accessible to the user.''

= Preliminary Results =
* Owner: [[User:Ian Spector | Ian Spector]]

''Initially, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T10:47:17Z

Eric Sodomka: /* Preliminary Results */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]], [[User:Jon Ericson | Jon Ericson]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Gideon Goldin | Gideon Goldin]], [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== Cognitive/HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

=== Fitts's Law ===

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Interaction Prediction ===
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

= Preliminary Results =
* Owner: [[User:Ian Spector | Ian Spector]]

''Initially, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T10:46:41Z

Eric Sodomka: /* Preliminary Results */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]], [[User:Jon Ericson | Jon Ericson]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Gideon Goldin | Gideon Goldin]], [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== Cognitive/HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

=== Fitts's Law ===

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Interaction Prediction ===
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

= Preliminary Results =

''Initially, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T10:46:28Z

Eric Sodomka: /* Preliminary Results */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]], [[User:Jon Ericson | Jon Ericson]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Gideon Goldin | Gideon Goldin]], [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== Cognitive/HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

=== Fitts's Law ===

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Interaction Prediction ===
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

= Preliminary Results =

''To start, let's make up some fictional (but reasonable) preliminary results that we'd like to see and think we can accomplish before submitting the proposal.''

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T10:44:31Z

Eric Sodomka: /* [Criticisms] */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]], [[User:Jon Ericson | Jon Ericson]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Gideon Goldin | Gideon Goldin]], [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== Cognitive/HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

=== Fitts's Law ===

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Interaction Prediction ===
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

= Preliminary Results =

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''Any criticisms or questions we have regarding the proposal can go here.''

CS295J/Research proposal (draft 2)

2009-04-21T10:36:56Z

Eric Sodomka: /* Interruptions */

= Introduction =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

We propose a framework for interface evaluation and recommendation that integrates behavioral models and design guidelines from both cognitive science and HCI. Our framework behaves like a committee of specialized experts, where each expert provides its own assessment of the interface, given its particular knowledge of HCI or cognitive science. For example, an expert may provide an evaluation based on the GOMS method, Fitts's law, Maeda's design principles, or cognitive models of learning and memory. An aggregator collects all of these assessments and weights the opinions of each expert, and outputs to the developer a merged evaluation score and a weighted set of recommendations.

Systematic methods of estimating human performance with computer interfaces are used only sparsely despite their obvious benefits, the reason being the overhead involved in implementing them. In order to test an interface, both manual coding systems like the GOMS variations and user simulations like those based on ACT-R/PM and EPIC require detailed pseudo-code descriptions of the user workflow with the application interface. Any change to the interface then requires extensive changes to the pseudo-code, a major problem because of the trial-and-error nature of interface design. Updating the models themselves is even more complicated. Even an expert in CPM-GOMS, for example, can't necessarily adapt it to take into account results from new cognitive research.

Our proposal makes automatic interface evaluation easier to use in several ways. First of all, we propose to divide the input to the system into three separate parts, functionality, user traces and interface. By separating the functionality from the interface, even radical interface changes will require updating only that part of the input. The user traces are also defined over the functionality so that they too translate across different interfaces. Second, the parallel modular architecture allows for a lower "entry cost" for using the tool. The system includes a broad array of evaluation modules some of which are very simple and some more complex. The simpler modules use only a subset of the input that a system like GOMS or ACT-R would require. This means that while more input will still lead to better output, interface designers can get minimal evaluations with only minimal information. For example, a visual search module may not require any functionality or user traces in order to determine whether all interface elements are distinct enough to be easy to find. Finally, a parallel modular architecture is much easier to augment with relevant cognitive and design evaluations.

= Overview of Contributions =
* Owners: [[User:Adam Darlow | Adam Darlow]], [[User:Eric Sodomka | Eric Sodomka]]

'''Note: For reference, this is the aggregate set contributions from last week. Maybe we can edit/add/remove from this as needed.'''

* Design and user-study evaluation of novel techniques for collecting and filtering user traces with respect to user goals.
* Extensible, low-cost architecture for integrating pupil-tracking, muscle-activity monitoring, and auditory recognition with user traces in existing applications.
* System for isolating cognitive, perceptual, and motor tasks from an interface design to generate CPM_GOMS models for analysis.
* Design and quantitative evaluation of semi-automated techniques for extracting critical paths from an existing CPM_GOMS model.
* Novel algorithm for analyzing and optimizing critical paths based on established research in cognitive science.
* A design tool that can provide a designer with recommendations for interface improvements, based on a ''unified matrix'' of cognitive principles and heuristic design guidelines.
* The creation of a language for ''abstractly representing user interfaces'' in terms of the layout of graphical components and the functional relationships between these components.
* A system for ''generating interaction histories'' within user interfaces to facilitate individual and collaborative scientific discovery, and to enable researchers to more easily document and analyze user behavior.
* A system that takes user traces and creates a GOMS model that decomposes user actions into various cognitive, perceptual, and motor control tasks.
* The development of other evaluation methods using various cognitive/HCI models and guidelines.
* A design tool that can provide a designer with recommendations for interface improvements. These recommendations can be made for a specific type of user or for the average user, as expressed by a utility function.

= Background / Related Work =

= Methodology =
'''TODO: Add some intro paragraph here.'''

== Collecting User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

''Given an interface, our first step is to run users on the interface and log these user interactions. We want to log actions at a sufficiently low level so that a GOMS model can be generated from the data. When possible, we'd also like to log data using additional sensing technologies, such as pupil-tracking, muscle-activity monitoring and auditory recognition; this information will help to analyze the explicit contributions of perception, cognition and motor skills with respect to user performance.''

== Generalizing User Traces ==
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

== Evaluation and Recommendation via Modules ==
* Owners: [[User:E J Kalafarski | E J Kalafarski]], [[User:Jon Ericson | Jon Ericson]]

''This section describes the aggregator, which takes the output of multiple independent modules and aggregates the results to provide (1) an evaluation and (2) recommendations for the user interface. We should explain how the aggregator weights the output of different modules (this could be based on historical performance of each module, or perhaps based on E.J.'s cognitive/HCI guidelines).''

== Sample Modules ==

=== CPM-GOMS ===
* Owners: [[User:Gideon Goldin | Gideon Goldin]], [[User:Steven Ellis | Steven Ellis]]

''This module will provide interface evaluations and suggestions based on a CPM-GOMS model of cognition for the given interface. It will provide a quantitative, predictive, cognition-based parameterization of usability. From empirically collected data, user trajectories through the model (critical paths) will be examined, highlighting bottlenecks within the interface, and offering suggested alterations to the interface to induce more optimal user trajectories.''

=== Cognitive/HCI Guidelines ===
* Owner: [[User:E J Kalafarski | E J Kalafarski]]

=== Fitts's Law ===

''This simple module will use Fitts's Law to provide interface evaluations and recommendations.''

=== Interruptions ===
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]

''While most usability testing focuses on low-level task performance, there is also previous work suggesting that users also work at a higher, working sphere level. This module attempts to evaluate a given interface with respect to these higher-level considerations, such as task switching.''

=== Interaction Prediction ===
* Owner: [[User:Trevor O'Brien | Trevor O'Brien]]

= Preliminary Results =

= [Criticisms] =
* Owner: [[User:Andrew Bragdon | Andrew Bragdon]]