Web Analytics Shootout Highlights the Importance of a Quality Controlled Environment
Posted by dpascoe on October 8, 2007
Eric Enge at Stone Temple Consulting published a paper in August titled “2007 Web Analytics Shootout“.
An interesting read - it was great that so many vendors participated. The paper is very helpful in describing how data from different vendors can vary.
The results are presented with confidence borne from the method used - a scientific method: 1) they established a quality-controlled environment; 2) they verified appropriate accuracy and precision of measurement devices; 3) they collected and analyzed data, formulate and test hypotheses, draw conclusions, drive decisions and actions. And herein lies the issue - very few companies are operating in a quality-controlled environment. This is even acknowledged in the study - here’s the excerpt:
“4. Realize that the biggest sources of error are JavaScript implementation errors. This could be as simple as pages that are missing the JavaScript, pages with malformed JavaScript, or problems that crop up as pages get added to the web site, moved, or removed from the web site.
This is an error completely within your control, and one that is quite potentially more devastating than any variance in the counting techniques used by the packages.“
That last sentence really packs a punch:
“this error is completely within your control”.
Yes it is, however not without automation. Sites are becoming larger, more complex, with more user contributors. They are changing more frequently, targeted at servicing even broader groups. Companies that have made the move to outsourced web analytics are also implementing some of the most complex development methods on the web - flash, ajax, and dynamic, client-side javascripting, for example. Without automated validation, the web analysts have no hope of staying on top of their implementation. This has given rise to the “accuracy doesn’t matter” myth.
Implementing a web analytics solutions with no automated way to validate it is like putting the web analyst in the space shuttle without any instruments. Try telling him or her accuracy doesn’t matter in that environment. 
Next phrase:
“one that is quite potentially more devastating than any variance in the counting techniques used by the packages.”
More than “quite potentially” - it is “most definitely” more devastating. Because humans perform the implementations, human error can and does creep in. The reasons listed in the study, in one single sentence, describe conditions we see every day, that can and do go undetected, and that have a direct and profound impact on data quality. Missing or malformed javscript introduces a systemic error, meaning that the data for those pages is missing 100% of the time.
Imagine that you have a website consisting of 10 pages - you want to collect traffic data for all 10.
- Unbeknown to you, one of the pages that is linked to is actually missing. No traffic is recorded for that page because it is never visited. That leaves 9.
- One page has a character in it that is causing the web analytics javascript not to function. The javascript is on the page; you think everything is fine, but no traffic is recorded for that page. That leaves 8.
- One page has the parameters set incorrectly so that when data about the page is transmitted it appears that a different page is actually getting the traffic intended for it. That leaves 7.
- One page is so heavy that many people leave before the tagging code, located at the bottom, has time to load. The data collected for this page is flawed.
- One page is missing from your internal search engine, so even if it contains the information that would perfectly match what the visitor wants, they can’t find it - if they do get to this page, it was through site navigation - the data collected for this page is flawed.
Now, you’re getting no data on 3 of the pages, and partial data on two - 50 percent of the pages have inherent problems that are directly impacting your data, and you are expecting to - and being expected to - draw conclusions, make recommendations and take action based on this information?
Without automated validation, you don’t know what you don’t know. Eric and the people who conducted the Shootout understood clearly the need to quality-control the environment. Although it is covered by only three sentences, this factor is as important as the outcome of the study.
