Measurements must Matter
Entry question: When should we try to measure something?
Rough Notes
Measurements must matter to decisions.
Decisions are interventions (on certain variables?). They must somehow cause a change in your terminal variables. (Else, why do them?) For example, you choose between Chinese food or pizza for the variable “dinner”, and you do this because you expect that it will affect your terminal variable “hunger” (and also “taste”, “money in wallet”, and “weight”).
My conclusion: We get useless research when we just focus on seeking the truth. We get measurements that don’t matter to any (“realistic”) decision. Examples? Well, string theory, I guess. You fail when you talk about variables that don’t really affect any realistic decision.
When some information can’t change any decision of yours, it holds no value for you. I could tell you the exact number of atoms in that lemon juice you’re drinking or its precise temperature but you won’t pay me anything for that information because it doesn’t matter - you’re going to drink and enjoy the lemon juice anyway. However, if you’re buying the juice from a dodgy-looking stall and I could tell you exactly what germs are there in your glass, you would likely pay for that information, because if it contained typhoid bacteria you would throw away that juice.
The more interesting failure mode is when you don’t change any decision that matters right now. Like willpower coefficient is 0.5 or 0.7 - does it matter to what you’re going to do? In essence, I think, why study this instead of something else that would matter more to another decision? You have to justify your decision to measure or understand this part of the universe.
Old Stuff
What are measurements good for?
This is a whole deep subject and I plan to write more as I keep learning.
For now: Measurements have no meaning without the Decision they are intended to inform.
Something has to change based on the different values of the measurement. If you take the same decision no matter what the value is, then you need not have measured it at all.
The measurement must matter. Something should depend on it.
Why do I rant about this now?
Cos I was making the mistake above. I was “measuring” something. Everyday I was trying to note down certain statistics and doing a bit of book-keeping. I was also thinking about how to record better statistics and stuff.
What’s wrong with that? Isn’t it good to have data?
The problem was that nothing would ever come of it. In all likelihood, I would never even look at those numbers I was collecting.
They would most probably become out of date as I miss writing on a couple of days.
The statistics I care about might change, rendering all this work useless.
I might not know how to interpret this data usefully in the future (I don’t have much experience with statistical analysis).
Most important of all, most likely no decision of mine would ever change based on what I had recorded. If things were really bad, I’d probably feel it myself and take action without needing to look over this data. If things didn’t seem all that bad, even if these numbers said something different, I probably won’t have the motivation to do anything about it.
And it’s such a f*cking drag having to record statistics. It’s a major pain in the ass. You want to be doing anything but writing down crap about stuff every night.
Why do I do it then?
Why go to the lengths of writing down stuff that you never use, going by past experience?
I feel like I must record some data. It feels like the intellectual thing to do. You know collect data, maybe someday transform it into a graph, and then display the graph. Wow! Math. Intellectual stuff.
Also, on each occasion, I fool myself saying that this time I will surely use this data and not just let it collect dust like all the other data.
It’s like the “Before” photo people take before going on a radical weight loss program. If you do get 6-pack abs or lose 100 pounds, you definitely want to have the “Before” photo to contrast with your “After” photo and get lots of praise or likes or whatever it is that people get. I feel like I will regret my lack-of-stats-recording a lot if I get to the end of some epic project in the future and want to take a look back. “Oh! So sad that I didn’t note anything down in the past. Curses!” So, just to be safe, I convince myself to measure.
F*ck that! If the project I complete is really that Epic, I will have way bigger benefits to enjoy than some simple nostalgic crap. It’s easy to overstate the benefits of nostalgic data for our future selves, especially cos we don’t have any of the epic rewards right now and thus can’t see that this shit will pale in comparison.
So, that sentimental crap won’t fly. Measurements must matter, else they’re useless.
Corollary
If you do want to measure, make sure your measurements are ultra-simple. Anything complicated and you’re almost certainly wasting your time. You will either not be able to use the data meaningfully or it will be too much of a hassle to do the analysis or you will just ignore the data, in which case at least you won’t have spent so much time on it.
One Legitimate use: Motivation?
If the measurements will end up giving you a sense of progress or otherwise increase your Motivation level for the task, then the measurement might be justified.
It can feel good to see your study-graph going upward or your weight going down. But the costs of regular tedious measurement and book-keeping should not outweigh the motivational benefits.
How much motivation do we really get from looking at the data we have recorded?
There is no doubt that measurements can provide feedback. Especially in video games, you get to see your current progress in a variety of ways - your score, the distance you’ve covered, the goodies you collected, and so on. No wonder you have so much motivation to continue playing the game.
But the measurements there are taken automatically. You don’t have to raise a finger for that. You keep playing and the game keeps informing you of your progress. The problem with real life is that you have to sit and make those measurements. And that can turn into quite a chore. In fact, it can be much like a To-Do List. You have to do it everyday, else you’ll lose the continuity and momentum. And when you have to do something you don’t really like and risk losing a lot otherwise, you start becoming stressed very easily.
Why is it a burden?
Not cool. Something we did for our benefit is turning out to harm us. The burden of recording data constantly can dampen your motivation really quickly.
Why is it like that?
Well, for one, I feel like if I don’t take measurements everyday, the data won’t be accurate anymore. It feels like it will be for nought. You have to fight constantly to keep it up to date.
Helpful Measurements
Measurements record your progress wrt a certain plan. So, if you’re trying to lose weight through a weight-loss cardio plan, then you will measure how many minutes of cardio you did, your heart rate at various times, the number of calories you lost, etc.
You will also keep track of the overall indicators like weight and BMI and body fat % and stuff. But those aren’t as helpful as the direct measurements as per your plan. Why not? Because they don’t tell you what you should change.
Say you aren’t measuring the specific things, just overall weight. If your weight stays the same even after a month, what should you do? Should you spend more time doing cardio? Or make it more intensive and increase your heart rate? Or do a different type of cardio that burns more calories? How much more? How would you know? You aren’t measuring those things directly. You are flying blind. You have no clue what to change. You may think it was because you didn’t burn enough calories, but how many more should you burn to get results? You don’t know cos you didn’t measure it.
Burn them all! Burn as much necessary!
Nope. Not possible. Dude, it’s a miracle if someone even spends 30 minutes on cardio everyday. People are rushed for time and don’t particularly want to do more cardio.
Principle of Economics #2: People face tradeoffs.
Increasing the time on cardio has costs. You need good reason to believe that it will actually be worth it. But you don’t have that. You’re just shooting in the dark, hoping that it will make you lose weight.
Lesson: If you have a plan, you need to be measuring its output in such a way that you can make intelligent changes to the plan.
Emphasis on “intelligent”.
Your measurements need to tell you that if you aren’t getting the overall results you wanted, and if this particular value X is below a threshold, then you need to do more of the sub-plan that X measures.
In other words, measurements must matter to some decision. The more fine-grained the decision is, the better. A measurement that tells you that the plan is working or not is cool. But it is too heavy-handed. A measurement that tells you that some specific action is lacking is much better.
Software entropy
Information systems are like vegetables kept in the open - they rot with time. Change is the biggest enemy of people who deal with big systems (complex programs, bureaucracies, etc.). Things are rarely neat and simple.
We saw that good measurements help track which parts of our plan are working well and which parts are not. This means that your measurements are gonna increase in number (How exactly?) as your plans becomes more complex.
How can a plan become more complex? Change.
You will rarely come up with the right plan on the first try. Either the search space is really big (as it usually is with humans actions) and you can’t really hit the target in one go. Or, the plan is so big that you can’t even really conceive it fully in your head. You think you have it, but there are actually gaping holes in your plans. You will discover those holes only as you go about implementing your “plan” and running head-first into the wall of reality.
Which means your plan is going to change. Depend on it. Change is the only constant.
Your plan might change, and so the measurements have to change too, because they measure your performance wrt a plan. Which either means the past measurements are useless or they have to be upgraded to the new format - pain in the ass.
You might come across some better measurement.
You have to take pains to counter the entropy.
Wasted measurements
Measuring your weight again and again - What came of it? Did the measurements matter?
Measuring how much time you’ve spent in studies everyday or how much studies you did.
Seriously, wtf do you think you’re gonna do with it? In a month, when you’re getting poor test scores, are you gonna look back at your dear measurements and say, “Oh! I studied for 1.57 hours everyday instead of 3. THAT’S why I didn’t do too well”? Really? You already know that you’ve been goofing off. You don’t even need the stats to tell you that.
Those measurements didn’t matter at all.
Writing thoughts down in a daily diary or journal
If you’re writing for the psychological benefits it gives you then and there, cool.
But if you’re trying to note down information, think hard. Does it matter? Can it possibly change any decision in the future?
Takeaway
If you’re ever thinking of making a measurement, think about the decision that it will possibly affect. If there is none, or if you don’t have any clue how it could change the decision, think again. Make measurements that matter.
comments powered by Disqus