Great post - if anything, I think this problem goes way beyond just real-time dashboards to extend to the vast majority of analysis done today (both by web and more traditional companies).
The statement that particularly resonated for me was [figure out] "if you’re looking at stats now because you’re curious and impatient, or because those stats will actually drive business decisions", but I'd take it one step further.
Analysis and stats are incredibly valuable when:
1) they are applied to a real business decision that is tied to actual value
2) you are willing to change what you are doing based on the result
3) you don't already know the answer
4) you have sufficient confidence in the result to act on it.
We need to get more used to stopping and thinking before we start analysis by laying out the decision we need help making, the different paths we're willing to take, and the amount of confidence we'll need to change our decisions. I could definitely be a lot more disciplined about it.
(I actually wrote a blog post last week on this same topic that lays out the above criteria in more detail and might be useful - though that was a reaction to the profusion of ads out there calling Big Data and Analytics "hot", that ignore how they actually drive real value. Blog post is here: http://www.klaviyo.com/blog/2012/07/16/the-curse-analytics-b...)
The statement that particularly resonated for me was [figure out] "if you’re looking at stats now because you’re curious and impatient, or because those stats will actually drive business decisions", but I'd take it one step further.
Analysis and stats are incredibly valuable when:
1) they are applied to a real business decision that is tied to actual value
2) you are willing to change what you are doing based on the result
3) you don't already know the answer
4) you have sufficient confidence in the result to act on it.
We need to get more used to stopping and thinking before we start analysis by laying out the decision we need help making, the different paths we're willing to take, and the amount of confidence we'll need to change our decisions. I could definitely be a lot more disciplined about it.
(I actually wrote a blog post last week on this same topic that lays out the above criteria in more detail and might be useful - though that was a reaction to the profusion of ads out there calling Big Data and Analytics "hot", that ignore how they actually drive real value. Blog post is here: http://www.klaviyo.com/blog/2012/07/16/the-curse-analytics-b...)