When the user first opens the homepage, their session ID and the current time are logged in a file. Let's say that the user enters "Stanford University" as their search.
The user is then redirected to the results page, which is (at the current implementation) a randomized ordering of the first ten Google search results for the query.
The user's query is noted in the logs.
After the user completes their search task, they will be asked to complete a survey. One question I am interested in seeing the answer to is the amount of time users think they personally spend on relevant websites vs. irrelevant websites. I will then compare their perceived visit times with the actual visit times, to see how accurate (or not accurate) users are at judging their own search behavior.
In order to deal with the problem of not being able to track the user's behavior on external websites, I developed a workaround by logging the time at which the user clicks on a link on the results page and the time at which they return to the Zoogle results. In this way, I cannot track detailed information of navigation within the website (such as how many internal pages the user looked at), but I can estimate the total amount of time spent on the website. I am thus modifying my hypothesis to be using the time spent on websites as a measure of implicit feedback instead of the original plan of looking at the bounce rate. I think that this behavior can also be revealing and possibly even correlated with the bounce rate, even though that is no longer the goal of the experiment.
I will be running experiments on Amazon's Mechanical Turk, so I hope to collect at least 105 datapoints for a 95% confidence interval with a Cohen's d of 0.5. Collecting enough data by running experiments manually would have been a near impossible task otherwise.
- How should I ask users whether or not they judged a website to be relevant? If I ask them right after they finish viewing the site, it could be too distracting and possibly affect their search behavior. However, if I ask them after they have completed the experiment, it is very likely that they have forgotten which websites they looked at, much less than whether or not they considered the website to be relevant.
- Alternatively, if I ask simple enough questions, would it be a good idea to make the simplifying assumption that the last site visited in most relevant, and the previously visited sites were less or not relevant?
- Other feedback is also welcome!