FigureEnergy study preparation.
1 – Purposes:
By using FigureEnergy (FE) system, we want to test our assumption that there is a difference between using the annotation system and not using the annotation system in the weekly energy usage changes and in the energy knowledge gained after using the FE system. We apply two reward mechanisms to encourage participants during the experiments, and we want to see which reward mechanism is better.
2 – Rewards mechanism
We define:
- “Flat” reward: is to give participants a fix amount of money x if they manage to save y% compared to their prior usage
- “Linear” reward: is to give participants £1.00 per energy N KWh they save
3 - Measures
- Weekly/Daily energy usage
- How much knowledge the participants gain from their energy consumption changes. This is required participants to complete two questionnaires, which are “before the experiment” and “after the experiment”
4 – Participant’s conditions
- Commitment of using AlertMe non-stop during the experiment (i.e. not turning off AlertMe’s hub power, internet...)
- The main cable of the participant’s house is not too far away from the Ethernet. The maximum range between the hub and the Meter Reader is 20 metres and a signal can be detected through 2 walls. If the Signal Strength icon is flashing, it means that your Display has a weak connection with your Meter Reader. Therefore, you need to try to situate your Display closer to your Meter Reader.
- Participants should have previous computer experience.
- Participants should not take any vacation during the experiment
5 – Plans before experiments
- A clear instruction about the experiments
- A short information pack of energy consumption (1-2 pages) is to allow participants aware of energy saving. This approach can increase energy awareness to all participants and make a fairness
- Each experiment would be held for 1-2 months
- Two questionnaires need to be prepared for participants to do before and after the experiment. This way we can measure how much knowledge they gain after using the system.
6 – Experiments
Experiment 1:
In this experiment, we use a “flat” reward mechanism as a motivation key, which is to give participants a fix amount of money of £10.00 if they manage to save 10% compared to their prior weekly usage.
We assume that participants who use FigureEnergy system with the annotation feature will gain more energy knowledge and save more energy than the participants who use FE system without the annotation feature.
Experiment 2:
This experiment process is exactly similar to the experiment 1. We swap the “flat” reward mechanism to “linear” reward mechanism. The “linear” rewards mechanism is to give participants £1.00 per energy KWh they save, compared to their prior weekly usage.
-------------------------------------------------------------------------------------------------------------
When these experiments are finished, we can compare the results to make an insightful analysis.
Hi Henry, once again, overall very good start! As promised, but later than I hoped, here are some more specific comments. Whenever possible, I try to ask questions and let you elaborate on things rather than writing what I think.. Let's see how it goes (it's a bit of an experiment for me).
ReplyDeleteNote I need to break this down into multiple comments for a limitation of blogspot.com
1 – Purposes:
By using FigureEnergy (FE) system, we want to test our assumption that there is a difference between using the annotation system and not using the annotation system in the weekly energy usage changes and in the energy knowledge gained after using the FE system.
Rather than "assumptions" we should talk about "hypotheses" here. You wrote "there is a difference" it would be more clear if you explain more in detail what we expect the difference to be.
Also this sentence is a bit convoluted in terms of English.
Why "weekly" usage?
We apply two reward mechanisms to encourage participants during the experiments, and we want to see which reward mechanism is better.
Why 2 rewards mechanisms (at this stage)? Please explain.
2 – Rewards mechanism
[...]
Good! (but see above)
3 - Measures
[...]
Good!
We can also measure how much (frequency & duration) participants use each part of the system. Through logging on the server side. We can actually record all the interaction to be able to reconstruct it, as if we were observing the user when they were doing things.
We will hopefully have a good amount of user generated content: the annotations of events. These may be another important source of information, even though it will not help us in comparing the two conditions..
One tricky thing here is that the users engagement with the system will also depend on the system usability and the "UX" that it provides. We need to be sure that the system works really smoothly, otherwise people may stop using it simply because it does not work well or because it is cumbersome. I say this is "tricky" because what we want to test is the concept (annotation creates knowledge), while these issues are matters of implementation.
(continued)
ReplyDelete4 – Participant’s conditions
"Conditions" is not the right word here. People normally talk about "inclusion/exclusion" criteria for participation in the study (however this is not exactly the same as what you write..)
- Commitment of using AlertMe non-stop during the experiment (i.e. not turning off AlertMe’s hub power, internet...)
Good.
- The main cable of the participant’s house is not too far away from the Ethernet. The maximum range between the hub and the Meter Reader is 20 metres and a signal can be detected through 2 walls. If the Signal Strength icon is flashing, it means that your Display has a weak connection with your Meter Reader. Therefore, you need to try to situate your Display closer to your Meter Reader.
This is a tricky definition. Maybe we can think in terms of house/flat size? This would also be good to make the experiment more controlled. Eg we could only take people in 1 or 2 bedroom flats.. (but then sometimes in flats you have that the counter is downstairs..)
- Participants should have previous computer experience.
Normally I use a trick here: I advertise the study on email lists, so we already know for sure that we deal with people who use email and email lists..
- Participants should not take any vacation during the experiment
This sounds a bit weird.. Maybe we can say that we ask participants to use the system for at least N weeks? Does it matter if they take a week break?
5 – Plans before experiments
- A clear instruction about the experiments
-> "clear instructions"
They should also be concise.
- A short information pack of energy consumption (1-2 pages) is to allow participants aware of energy saving. This approach can increase energy awareness to all participants and make a fairness
Good point!
- Each experiment would be held for 1-2 months
- Two questionnaires need to be prepared for participants to do before and after the experiment. This way we can measure how much knowledge they gain after using the system.
Good.
We may also want to record one week before they start the study, this will serve multiple purposes: 1) let us test that the system works, 2) calculate the baseline consumption
6 – Experiments
Experiment 1:
In this experiment, we use a “flat” reward mechanism as a motivation key, which is to give participants a fix amount of money of £10.00 if they manage to save 10% compared to their prior weekly usage.
We assume that participants who use FigureEnergy system with the annotation feature will gain more energy knowledge and save more energy than the participants who use FE system without the annotation feature.
Sounds good! Not sure about the numbers, though.
Experiment 2:
This experiment process is exactly similar to the experiment 1. We swap the “flat” reward mechanism to “linear” reward mechanism. The “linear” rewards mechanism is to give participants £1.00 per energy KWh they save, compared to their prior weekly usage.
Well explained, but see above about why 2 reward mechamisms.
When these experiments are finished, we can compare the results to make an insightful analysis.
You did not mention interviews. When shall we interview people? What shall we ask?
I have rewrite in some parts:
ReplyDelete1 – Purposes:
FigureEnergy (FE) is a web-based system, which allows users to interact with their own energy consumption. In addition, FE offers an annotation system which encourages people to tag the segment of usages associated with their specific activities in the past.
We want to examine hypotheses about the use of the annotation system in our FE system. The alternative hypothesis states that there is a difference between using the annotation system and not using the annotation in the consumption of energy changes and in gaining knowledge of saving energy. We expect to see that, with the help of the annotation system, participants will have better understanding about their own energy consumption and are able to save more energy consumption than without the use of the annotation system.
2 – Rewards mechanism
We define:
- “Flat” reward: is to give participants a fix amount of money x if they manage to save y% compared to their prior usage
- “Linear” reward: is to give participants £1.00 per energy N KWh they save
We apply two reward mechanisms, such as “flat” reward and “linear” reward, to motivate participants during the experiments. They are two types of motivation which allow participants to challenge themselves in the use of their energy devices to get money. Typically the “flat” reward is fun, challenging goal settings and offering a learning curve and determination to participants to observe their own energy devices, while the “linear” reward is giving a simple path of motivation in saving energy.
These two reward mechanisms also allow us to observe more in the changes of behaviours from the participants.
4 – Participant’s inclusion/exclusion
- Commitment of using AlertMe non-stop during the experiment (i.e. not turning off AlertMe’s hub power, internet...)
- The main cable of the participant’s house is not too far away from the Ethernet. The maximum range between the hub and the Meter Reader is 20 metres and a signal can be detected through 2 walls. Therefore we can only take people in 1 or 2 bedroom flats for our experiments.
- Participants should have previous computer experience. (To test this, we might just send the advertisement on email lists and only deal with people who response from our email. We also might give a short paragraph (100 words) to the participants to type in Word document, therefore the qualified participants should type it less or equal than 5 mins.
- Participants should use the system for at least 3 weeks.
@Enrico: As our system has basically ready to test, therefore we can start next Monday. What do you think?
ReplyDelete