UX from 30,000ft: Lecture 15 - Designing Your Evaluations

specific: COMP33511

topic: Designing Your Evaluations

# {{name}}: {{topic}}

### Lecture 15 (50 minutes)

### [@sharpic](http://twitter.com/sharpic)

.controls[[SAQ](saqs.html) [D](discuss.html) [OH](oh.html) [C](coffee.html) [P](coffee.html#2) [SLIDES](http://sharpic.github.io/COMP33511/#slides) [↩](#)]

---

specific: COMP33511

topic: Part III: Validating the User Experience

# {{name}}

# {{topic}}

---

noteid: Ch 09 - Designing Your Evaluations

specific: COMP33511

website: http://sharpic.github.io/COMP33511

author: [@sharpic](http://twitter.com/sharpic)

---

topic: Designing Your Evaluations

.credits[ {{author}} | UX from 30,000ft | {{specific}} | {{topic}}]
.controls[[SAQ](saqs.html) [D](discuss.html) [OH](oh.html) [C](coffee.html) [P](coffee.html#2) [SLIDES](http://sharpic.github.io/COMP33511/#slides) [↩](#)]

## {{topic}}

1. Badly Designed  = Incorrect Analysis;

1. Incorrect Analysis = Incorrect Conclusions; which means

1. Success of your Interventions in Doubt.

### This Means

If evaluations are not designed correctly the previous ≈207 pages of the course notes have been, to a large extent, pointless.

---

topic: Science and Generalisation

## {{topic}}

### Inductive reasoning

Evaluates and then applies to the general 'population' abstractions of observations of individual instances of members of the same population

### Deductive reasoning

Evaluates a set of premises which then necessitate a conclusion -- for example: {(1) Herbivores only eat plant matter; (2) All vegetables contain only plant matter; (3) All cows are herbivores} ⇒ Therefore, vegetables are a suitable food source for Cows.}

1. Therefore, the conclusion must be true provided that the premises are true;

1. Note that you could not say 'Therefore, all cows eat vegetables' because fruit also contains only plant matter; as do grass and trees.

---

topic: Scientific Bedrock

## {{topic}}

### To Be Scientific

A method of inquiry must be based on the gathering of observable, empirical and measurable evidence, and be subject to specific principles of reasoning.

---

topic: Scientific Bedrock

## {{topic}}

An Inductive Example

1. Firstly, we create an hypothesis which, in the best case, cannot be otherwise interpreted and is 'refutable'; for example we might make the statement 'all swans are white'. In this case we may have travelled widely and tried to observe swans in every country and continent in an attempt to support our hypothesis.

2. While, we may be able to amass many observations of white swans we must also realise that a statement must be refutable. If the hypothesis remains intact it must be correct; in our example we may try to observe every swan that exists in, say, the UK, or Europe, or the Americas, which is not white.

3. However, one instance of an observation of a non-white swan will disapprove our hypothesis; in this case when we arrive in Australia we discover a black swan, in this case we can see all swans are not white and our hypothesis is found to be incorrect.

---

topic: Scientific Bedrock

## {{topic}}

1. Many debates regarding the question of whether inductive reasoning leads to truth;

1. We can make some inductive leaps if they are based on good science;

1. These leaps may not be absolutely accurate; but

1. May well assist us in our understanding; in the

1. UX domain we use **mathematical (statistical) methods** to help us understand these points.

---

topic: Mathematical (Statistical) Methods

## {{topic}}

1. Generalise results to enable us to say something about the wider population; so

1. We use well formed and tested statistical tests;

1. Which enables use to mathematically generalise to a population; this is called,

1. **External Validity**.

### No 100% Certainty

All we have is a level of confidence in how a particular test relates to the population, and therefore how useful the knowledge generated from it really is.

---

topic: Variables / UX

## {{topic}}

1. **Behavioural**: Equated to the user;

1. **Stimulus**: Equated to the interface or the computer system;

1. **Observable Response**: the thing we measure to understand if there is a benefit after we have manipulated the stimulus; and

1. **Subject**: Factors such as age, weight, gender.

- **Independent Variable**: The thing that we manipulate -- the lower the number of independent variables, the more confident we can be about the data collected and the results of the analysis; and

- **Dependent Variable**: The thing that we measure.

---

topic: Measuring Dependent Variables

## {{topic}}

1. **Nominal Variable** (plural nominal variables). A variable with values which have no numerical value, such as gender or occupation. For example: *opposite, alternate, whorled*. Also known as **Categorical Variable** (plural categorical variables).

1. **Ordinal Variable** (plural ordinal variables). A variable with values whose order is significant, but on which no meaningful arithmetic-like operations can be performed. For example: *fast < very fast < very, very fast*

1. **Interval Variable** (plural interval variables). An ordinal variable with the additional property that the magnitudes of the differences between two values are meaningful. For example: *10PM (today) > 8PM (today) --- 10PM (today) - 8PM (today) = 2 hours*.

1. **Ratio Variable** (plural ratio variables). A variable with the features of interval variable and, additionally, whose any two values have meaningful ratio, making the operations of multiplication and division meaningful. For example: *10 meters per second > 8 meters per second --- 10 mps - 8 mps = 2 mps*.

.aside[**Variables, and their measurement, are important.** They inform the experimental design process and the kind of analysis that will be possible once the data has been collected.]

---

topic: Measuring Dependent Variables

## {{topic}}

1. **Null Hypothesis**: Which dictates that there is no difference between two conditions beyond chance differences; or

1. **Hypothesis**: Which dictates there is a difference and supports the hypothesis proposed.

### Strong and Weak

A hypothesis must be 'strong' to be testable.

### Nothing is Ever Proved

Hypotheses are supported or disproved - NOT ever proved (in empirical work).. Why?