1 – What is statistical hypothesis testing?
A statistical hypothesis testing is a method of making decisions using data, directly tested through an empirical investigation. The hypothesis testing lays both the foundation of experiments and the basis of statistical significance testing.
Typically, the hypotheses of an experiment involve examining a relationship between the dependent variables and the independent variables. They specify the dependent variables to be observed and the independent variables to be controlled. An experiment normally has a null hypothesis and an alternative hypothesis. A null hypothesis (Ho) states that there is no difference between experimental treatments, while the alternative hypothesis (H1) is a statement that is mutually exclusive with the null hypothesis. The goal of the experiment is to test the null hypothesis against the alternative hypothesis and decide which one should be accepted and which one should be rejected, based on the results of any significance test. [1,2]
Significance testing technically is a process to determine the probability that the null hypothesis is true, where a null hypothesis contrasted with the alternative hypothesis. All significance tests are subject to two types of error. Type I errors refer to the situation when the null hypothesis is mistakenly rejected when it is actually true. Type II errors refer to the situation of not rejecting the null hypothesis when it is actually false. Type I errors generally are worse than Type II errors, therefore the alpha threshold to determine the probability of making Type I errors should be kept low, widely accepted at 0.05.
2 – What implications does it have on the design of our study?
We have several testing on our system to examine how effective our system design offer to users an ease of use in managing their energy consumption and saving their own usages. We can use the hypothesis testing on experimental research to make a decision on our study.
We might consider taking two experimental studies:
1 – With the use of activity annotation, users allow to tag a segment of usage from their previous energy consumption profile with specific activities. We will test to figure out whether or not using the “flat reward” (a fix amount of reward) better than using the “linear reward” (the more energy usages which users saved, the more rewards users offer)
2 – Without the user of activity annotation, We will test to figure out whether or not using the “flat reward” (a fix amount of reward) better than using the “linear reward” (the more energy usages which users saved, the more rewards users offer).
References
[1] Lazar J., Feng J. H., and Hochheiser H., “Research Methods in Human-Computer Interaction”
[2] Interaction Design – beyond human-computer interaction (corebook)
This comment has been removed by the author.
ReplyDeleteHi Henry. This is a good start!
ReplyDeleteHere are some comments and questions.
You wrote "examining a relationship between the dependent variables and the independent variables" -- can you define what are dependent variables and independent variables? Can you make some examples of each, for example from our study?
"to examine how effective our system design offer to users an ease of use in managing their energy consumption and saving their own usages" -- this is a bit vague. Given that it is a crucial matter, I think it would be useful if you try and clarify this. For example, what do you mean by "ease of use"? It may help to think about this in the context of the usability goals and user experience goals discussed in the 1st chapter of the "Interaction Design" book. You also state what are the hypotheses for the study, and what are the dependent and independent variables that we can use for each hypothesis that we want to test.
Finally, there are some issues with English language. I am not sure what is the best way to tackle those, let's discuss this in the meeting.
Independent variables refer to the factors that the researchers are interested in studying. Dependent variables refer to the outcome or effect that the researchers are interested in. In experiments, researchers primarily want to study the relationship between dependent variables and independent variables. More specifically, the researcher wants to find out whether and how to changes in independent variables induce changes in dependent variables.
ReplyDeleteFor example, we want to find out the difference between allowing users to use previously annotated activities and not allowing users to use annotated activities in the amount of energy reduction weekly. The independent variable is the activity annotation function (to use or not to use). The dependent variable is the amount of energy deduction weekly. During the experiment, we have full control to randomly assign each participant to an experimental condition. In contrast, the amount of energy reduced weekly by participants is highly dependent on individual behavioural factors that we cannot control. Some participants will be saved more energy than others due to a number of factors, such as the use of activities annotation, energy awareness, previous computer experience, physical capabilities and so on.
@Enrico: I think I need to discuss to you more about our study later when you get well.
Thanks for expanding on this Henry.
ReplyDeleteYou are on a good track.
You are right in saying that "the amount of energy reduced weekly by participants is highly dependent on individual behavioural factors", but there are some other things to consider.
As we discussed in a meeting few weeks ago, we can influence (perhaps not quite control, but..) this by designing reward mechanisms -- e.g. give participants £1 per every N KWh they save, or give them £20 if they manage to save x% compared to their prior usage.
Moreover, we discussed about another measure: rather than measuring if and how much participants save, we can measure how much their knowledge of their consumption changes. One way of doing this would be to get them answer two questionnaires (think of them as "exams"), one before and one after the study, and see if they perform better after the study in the condition with the annotation system.
Finally, you mention "energy awareness, previous computer experience, physical capabilities" -- these are conditions that can be influenced by the way we recruit participants. For example, in the past I designed studies where participants needed not to use eye-glasses (but contact lenses were ok) or to be able to walk for 20 minutes.. this is different than what we have, but hope it will give you ideas.