DataAnalysis 0

Prévia do material em texto

CCU 2010 / 2011 
Lesson 10 
Data Analysis, Interpretation and 
Presentation 
©2008-2010 IST & RP 1 
Previous Lesson (1) 
  Usability 
  Effectiveness 
  Efficiency 
  Satisfaction 
  Ease to use and learn 
  User acceptance 
©2008-2010 IST & RP 2 
Previous Lesson (2) 
  Usability Engineering 
  Define usability goals 
  Heuristics 
  Is part of the requirements engineering 
  How to test / validate implementation 
  Test 
  Usability is determined by measuring the users' 
interaction with the product 
©2008-2010 IST & RP 3 
Previous Lesson (3) 
  Goals 
  Quantitative 
  Absolute 
  Relative 
  Subjective 
  Qualitative 
  Measured indirectly through quantitative goals 
©2008-2010 IST & RP 4 
Previous Lesson (4) 
  Specifying Goals / Usability Tests 
  Attribute 
  Measure 
  Method of Measure 
  4 reference levels 
  Current value 
  Minimum (Acceptable) 
  Target 
  Optimum 
©2008-2010 IST & RP 5 
Previous Lesson (5) 
  Measures 
  Time, number of errors, operations, etc. 
  Reactions, opinions, preferences, satisfaction, 
etc. 
  Methods of Measure 
  Must be contextualized 
  Define specific tasks with specific arguments 
©2008-2010 IST & RP 6 
  Introduction 
  Quantitative analysis 
  Qualitative analysis 
  Tools 
  Theoretical frameworks 
  Presenting the findings 
©2008-2010 IST & RP 7 
Quantitative and Qualitative Data 
  Quantitative data 
  Can be easily translated into numbers 
  Qualitative data 
  Difficult to measure/count 
  All information capture methods discussed may 
create qualitative or quantitative data 
  Any qualitative data can be translated into 
numbers 
  With more or less trouble... 
©2008-2010 IST & RP 8 
Use and Abuse of Figures 
  Numbers usually present clearer results 
  However, they can lead to misinterpretation 
  Ex: analysing interviews regarding a new 
product. Count the number of times the product 
is mentioned. 
  Ex: stating the 50% of users are happy with the 
product when only 4 users were inquired 
  Always present the context! 
©2008-2010 IST & RP 9 
The First Steps 
  Transcribe interviews 
  Find the most relevant / important parts 
  Questionnaires / closed questions 
  Remove false answers 
  Filter data into sets 
  Ex: segment by age 
  Observation 
  Synchronize data 
©2008-2010 IST & RP 10 
Summary 
Raw data Example Qualitative 
Example 
Quantitative Initial steps 
Interviews Audio, notes video 
Responses to 
open questions Age, job Transcription 
Questionnaires Written, online database 
Responses to 
open questions 
Years of 
experience 
Clean up data 
Cluster data 
Observation 
Notes, photos, 
audio, video, 
data logs, 
written 
Descriptions of 
behaviours 
Time spent on a 
task 
Transcription 
Synchronization 
©2008-2010 IST & RP 11 
  Introduction 
  Quantitative analysis 
  Qualitative analysis 
  Tools 
  Theoretical frameworks 
  Presenting the findings 
©2008-2010 IST & RP 12 
Simple Quantitative Analysis 
  Use averages and percentages 
  Three types of averages 
  Mean, Median, Mode 
  Ex: Data: {2,3,4,6,6,7,7,7,8} 
  Median = 6, Mode =7, Mean = 5,56 
  Ex: Data: {2,2,2,2,450} 
  Present standard deviation 
©2008-2010 IST & RP 13 
Simple Quantitative Analysis 
  Place data in tables 
  Distinguish “don't know” from “did not answer” 
  How to present enumerations? 
User Job 
1 Teacher 
2 Driver 
User Teacher Driver Other 
1 X 
2 X X 
©2008-2010 IST & RP 14 
Questions Influence the Type of 
Analysis 
  “How do you felt about the system?” 
  If the sample is too big analysing this can be 
a problem 
  Many different answers 
  It will be hard to summarize 
©2008-2010 IST & RP 15 
Questions Influence the Type of 
Analysis 
  “In your opinion the system was amusing, 
irritating or neither?” 
  14 out of 26 (54%) found the system 
amusing 
User Amusing Irritating Neither 
A 1 
B 1 
... 
Z 1 
Total 14 5 7 
©2008-2010 IST & RP 16 
Questions Influence the Type of 
Analysis 
  “In your opinion the system was amusing:” 
  1) Strongly agree; 2) Agree; 3) Neither; 4) Disagree; 5) Strongly disagree 
  4 out of 26 (15%) disagree that the system 
is amusing 
User SA A N D SD 
A 1 
B 1 
C 1 
... 
Z 1 
Total 5 7 10 1 3 
©2008-2010 IST & RP 17 
Simple Quantitative Analysis 
  Find Outliers 
  Values that are significantly different 
  May represent noise in the data 
  Study / analyze these special cases 
©2008-2010 IST & RP 18 
Comparing Two Systems 
  Bar graphs help comparing 
Was the system helpful? 
©2008-2010 IST & RP 19 
Example 
  Online computer game 
  Start Wars Galaxy 
©2008-2010 IST & RP 20 
Example - SWG 
  Goal: Identify patterns of interaction 
  Data logs, in-game videos and ethnography 
  Data from 26 days (21 hours a day) 
  5493 unique players 
  Two important locations 
  Starport and Cantina (Coronet City) 
©2008-2010 IST & RP 21 
Example - SWG 
  Interactions of two main categories 
  Gestures (e.g. smile, greet, clap) 
  Public messages 
Gesture Cantina (%total) Gesture 
Starport 
(%total) 
Smile 18,13 Thank 15,95 
Cheer 9,57 Bow 12,29 
Clap 7,77 Wave 9,81 
Wave 6,27 Flail 8,17 
Wink 4,22 Smile 7,89 
Grin 3,72 Nod 7,03 
Nod 3,23 Salute 2,48 
Bow 3,22 Pet 1,95 
©2008-2010 IST & RP 22 
Example - SWG 
©2008-2010 IST & RP 23 
Example - SWG 
  Gestures made and received - Cantina 
  Size of circle → number of messages 
©2008-2010 IST & RP 24 
Example - SWG 
  Gestures made and received - Starport 
©2008-2010 IST & RP 25 
Example 
  Some Results 
  Most of the players do not interact much 
  Some players send many messages but do not use 
gestures 
  In the cantina 
  On average 1 gesture made, 1 gesture received 
and 4 messages exchanged 
  In the Starport 
  Few gestures but more public messages – it is 
the trading spot 
©2008-2010 IST & RP 26 
Qualitative Analysis: Remarks 
  If you only have a few records it is more 
important to analyse the individual than 
trends 
  Show tabular data 
  Don't throw away the raw data 
©2008-2010 IST & RP 27 
  Introduction 
  Quantitative analysis 
  Qualitative analysis 
  Tools 
  Theoretical frameworks 
  Presenting the findings 
©2008-2010 IST & RP 28 
Qualitative Analysis 
  Using frameworks creates structured data 
  Ex: observation frameworks 
  Three simple methods 
  Identifying recurring patterns 
  Categorizing data 
  Analyzing critical incidents 
©2008-2010 IST & RP 29 
Identifying Recurring Patterns 
  Goals of the study provide orientation 
  Major themes and minor themes 
  Ex: Study the usability of train travel website 
  Major theme: comment on the station stops 
  Minor theme: comment on the company logo 
  Keep a description of themes 
  Define its granularity 
  Refine the list, but not too much 
©2008-2010 IST & RP 30 
Identifying Recurring Patterns 
  Use affinity Diagrams 
©2008-2010 IST & RP 31 
Categorizing Data 
  Tag categories to descriptions 
  Words, sentences, paragraphs 
  Use categorization criteria 
  May be based on a scheme 
  May emerge from recurring patterns (themes) 
  Test the emergent scheme properly 
  Give the scheme to different people 
  Compute the inter-rater reliability (% of agreement) 
©2008-2010IST & RP 32 
Categorizing Scheme (Rens, 1997) 
  Interface Problems 
I1 - Show evidence of Dissatisfaction 
I2 - Show evidence of Uncertainty 
I3 - Show evidence of Surprise 
I4 - Show evidence of Physical Discomfort 
I5 - Show evidence of Fatigue 
I6 - Show evidence of Difficulty in Seeing 
particular aspects 
©2008-2010 IST & RP 33 
Categorizing Scheme (Rens, 1997) 
  Interface Problems 
I7 - Show evidence of Having Problems Achieving 
a Goal 
I8 - Show evidence that the user Made an Error 
I9 - Show evidence that the user was Unable to 
Recover from an Error 
I10 - Suggestions made by the user 
©2008-2010 IST & RP 34 
Categorizing Scheme (Rens, 1997) 
  Content Problems 
C1 - Show evidence of Dissatisfaction 
C2 - Show evidence of Uncertainty 
C3 - Show evidence of Misunderstanding 
C4 - Suggestions made by the user 
©2008-2010 IST & RP 35 
Example 
 I'm thinking that it it's just a lot of information to 
absorb from the screen. I just don't concentrate 
very well when I'm looking at the screen. I have a 
very clear idea of what I've read so far... but it's 
because of the headings. … It would still be nice to 
see it on a piece of paper, because it is a lot of text 
to read. … There is so much reference to all those 
previously said. ... 
©2008-2010 IST & RP 36 
Example 
 [I'm thinking that it it's just a lot of information to 
absorb from the screen I1]. [I just don't 
concentrate very well when I'm looking at the 
screen I1]. I have a very clear idea of what I've read 
so far... [but it's because of the headings I1]. … [It 
would still be nice to see it on a piece of paper I10], 
[because it is a lot of text to read I1]. … [There is 
so much reference to all those previously said C1]. 
… 
©2008-2010 IST & RP 37 
Categorizing Data 
  The criteria may evolve 
  May be refined through iteration 
  Video descriptions can also be tagged 
  More challenging! 
  Count the categories and create tables 
similar to quantitative analysis 
©2008-2010 IST & RP 38 
Looking for Critical Incidents 
  Identify areas to explore 
  Analysing all data is very time consuming and 
often unnecessary 
  Identify “good” and “bad” incidents 
  Two principles: 
  Report facts rather than general impressions 
  Reports should be limited to behaviours that 
make a significant contribution 
©2008-2010 IST & RP 39 
Looking for Critical Incidents 
  Critical Incidents are Situations that: 
  Provoke silence 
  Provoke puzzlement 
  Provoke user comments 
  … 
  Example: 
  “On one journey the system gave directions to 
turn right when the destination was to the left.” 
©2008-2010 IST & RP 40 
Qualitative Analysis 
  Combine techniques 
  Common approach 
1.  Find critical incidents 
2.  Identify recurring patterns 
3.  Categorize data 
©2008-2010 IST & RP 41 
  Introduction 
  Quantitative analysis 
  Qualitative analysis 
  Tools 
  Theoretical frameworks 
  Presenting the findings 
©2008-2010 IST & RP 42 
Tools 
  Spreadsheets (e.g. MS Excel) 
  Statistics (e.g. SPSS) 
  Qualitative analysis tools 
  Categorization and theme-based analysis 
  Quantitative analysis of text-based data 
©2008-2010 IST & RP 43 
Tools 
  CAQDAS Networking Project, based at the 
University of Surrey 
 http://caqdas.soc.surrey.ac.uk/ 
  Examples 
  N6 (formerly NUD*IST) 
  NVIVO 
  ANVIL - free 
  Video annotation 
  Weft QDA - free 
©2008-2010 IST & RP 44 
Tools 
  Weft QDA 
©2008-2010 IST & RP 45 
Tools 
  ANVIL 
©2008-2010 IST & RP 46 
  Introduction 
  Quantitative analysis 
  Qualitative analysis 
  Tools 
  Theoretical frameworks 
  Presenting the findings 
©2008-2010 IST & RP 47 
Theoretical Frameworks 
  Data analysis based on theoretical 
frameworks provides further insight 
  Three such frameworks are: 
  Grounded Theory 
  Distributed Cognition 
  Activity Theory 
©2008-2010 IST & RP 48 
Grounded Theory 
  Aims to derive theory from systematic analysis 
of data 
  Inverse “scientific method” 
  Based on categorization approach (called here 
coding) 
  Three levels of coding 
  Open: identify categories 
  Axial: elaborate and find relations 
  Selective: form theoretical scheme 
  Core categories 
©2008-2010 IST & RP 49 
Grounded Theory 
  Analytic tools 
  Questioning 
  The data 
  Comparisons 
  Between objects of categories 
  Reveals different perspectives 
  Evolve the model 
  Ex: merge categories 
©2008-2010 IST & RP 50 
Distributed Cognition 
  The people, environment and artefacts are 
regarded as one cognitive system 
  Used for analyzing collaborative work 
  Focuses on information propagation and 
transformation 
  Representational states across media 
  Internal 
  Ex: People’s memory 
  External 
  Ex: Notes, charts 
©2008-2010 IST & RP 51 
Distributed Cognition 
  Example: 
  Call Centre 
  Artefacts 
  Telephone 
  Paper 
  Catalogue 
  System 
  Shared 
©2008-2010 IST & RP 52 
Distributed Cognition 
  Example – Employment Verification 
1.  Telephone 
2.  User's working memory 
3.  Catalogue call 
4.  Paper 
  Person details. Ex: Social security number 
5.  System – Data Base 
6.  Paper 
  Required information 
7.  Caller memory 
©2008-2010 IST & RP 53 
Activity Theory 
  Explains human behaviour in terms of our 
practical activity with the world 
  Two key models: 
  Outlines what constitutes an ‘activity’ 
  Describes the mediating role of artefacts 
©2008-2010 IST & RP 54 
Activity Theory 
©2008-2010 IST & RP 55 
Activity Theory 
  Operations - Conditions 
  Routine behaviours 
  Little conscious attention 
  Ex: typing 
  Actions - Goals 
  Conscious planning 
  Ex: writing a glossary 
©2008-2010 IST & RP 56 
Activity Theory 
  Activity - Motives 
  Context to understand the actions 
  Ex: writing an essay 
  Dynamics of the system 
  An action may become an operation 
  It becomes routine 
  An operation may become an action 
  Find an obstacle 
  An activity may become an action, and so on... 
©2008-2010 IST & RP 57 
Activity Theory 
Engeström, 1999 
©2008-2010 IST & RP 58 
Activity Theory 
  Identify tensions between components 
  Examples: 
  Community limits communication? 
  Tool enables access to required knowledge? 
  Division of labour keeps workload balance? 
  Conflicting rules for rating documents? 
©2008-2010 IST & RP 59 
  Introduction 
  Quantitative analysis 
  Qualitative analysis 
  Tools 
  Theoretical frameworks 
  Presenting the findings 
©2008-2010 IST & RP 60 
Presenting the Findings 
  Only make claims that your data supports 
  The best way to present your findings depends on the 
audience, the purpose, and the data gathering and 
analysis undertaken 
  Graphical representations (see above) may be 
appropriate for presentation 
  Other techniques are: 
  Rigorous notations, e.g. UML 
  Using stories, e.g. to create scenarios 
  Summarizing the findings 
©2008-2010 IST & RP 61