Toggle the Factor you're Curious about
The Most Basic Test
There was an old story passed down among scientists, a cautionary tale, the story of Blondlot and the N-Rays.
Shortly after the discovery of X-Rays, an eminent French physicist named Prosper-Rene Blondlot - who had been first to measure the speed of radio waves and show that they propagated at the speed of light - had announced the discovery of an amazing new phenomenon, N-Rays, which would induce a faint brightening of a screen. You had to look hard to see it, but it was there. N-Rays had all sorts of interesting properties. They were bent by aluminium and could be focused by an aluminium prism into striking a treated thread of cadmium sulfide, which would then glow faintly in the dark…
Soon dozens of other scientists had confirmed Blondlot’s results, especially in France.
But there were still other scientists, in England and Germany, who said they weren’t quite sure they could see that faint glow.
Blondlot had said they were probably setting up the machinery wrong.
How would you test Blondlot’s theory about N-rays?
He’s claiming that if you have an aluminium prism and a treated thread of cadmium sulfide, you will get a faint glowing in the dark. In other words, he’s saying that aluminium prism and cadmium sulfide are relevant factors for the output variable of a faint glowing.
If the aluminium prism is truly a relevant factor, then changing it should result in some change in the output variable (for at least some baselines). So, as an initial test, remove it and see what happens.
One day Blondlot had given a demonstration of N-Rays. The lights had turned out, and his assistant had called off the brightening and darkening as Blondlot performed his manipulations.
It had been a normal demonstration, all the results going as expected.
Even though an American scientist named Robert Wood had quietly stolen the aluminium prism from the center of Blondlot’s mechanism.
And that had been the end of N-Rays.
Robert Wood removed the aluminium prism and Blondlot still got the same result. So, the aluminium prism wasn’t really a relevant factor for a change between a faint glowing and no faint glowing! That was a pretty big blow to the N-rays theory, which said that the aluminium prism was relevant.
Hypothesis: Suppose you’re told that an input factor is relevant to the output of a particular input. If you toggle that input factor, you can see if it has any impact on the output. If you don’t toggle it, you won’t know if it has any effect at all.
Therefore, if you’re curious whether a factor is relevant, toggle it.
Toggling doesn’t have to be as extreme as removing the factor entirely. You can change its value in degrees. For example, if the aluminium prism is one foot away from the cadmium sulfide, move it two feet away. If you still get the exact same result, you should be suspicious. But you can still try moving it ten feet away. If you get the same result again, you should be even more suspicious. Move it out of the room entirely and dismiss the theory if you get the same result. You can also toggle the type of material, such as a copper prism instead of an aluminium prism, or the shape of the material, such as a cube instead of a prism.
Notice how we didn’t know any details about Blondlot’s model. All we knew was that the aluminium prism and cadmium sulfide were relevant factors. Even just that little bit of information was enough for us to test the theory. The faint glowing didn’t change after we removed the prism and so we rejected it as a relevant factor for that baseline. If the faint glowing had changed after removing or shifting the aluminium prism, then we would have accepted that the aluminium prism was a relevant factor.
So, the bare minimum test for proposing a model is showing that the factors we say are relevant are actually relevant. We do this by toggling each of them from some example and showing that the output changes. We should not claim some factor is relevant without a toggled example showing its relevance. For example, when Paul Graham wrote that startup founders must be relentlessly resourceful, he toggled the “resourceful” part to show that just being relentless wasn’t enough:
I was writing a talk for investors, and I had to explain what to look for in founders. What would someone who was the opposite of hapless be like? They’d be relentlessly resourceful. Not merely relentless. That’s not enough to make things go your way except in a few mostly uninteresting domains. In any interesting domain, the difficulties will be novel. Which means you can’t simply plow through them, because you don’t know initially how hard they are; you don’t know whether you’re about to plow through a block of foam or granite. So you have to be resourceful. You have to keep trying new things.
– Relentlessly Resourceful, Paul Graham
This technique works for any diff, not just for a model like that of Blondlot’s. Suppose you see some change in input that produces a change from the default outcome. Describe that change in input in as much detail as possible. For example, your friend’s car is faster than yours and it has a different brand of tires and a different brand of fuel. You will probably get a pretty large diff with lots of potentially relevant factors. Toggle those factors and slowly shorten that list. So, toggle the tires on your car to see if your car gets any faster. If not, you can tell that the brand of tires wasn’t relevant.
A caveat about toggling factors to test their relevance is that a factor may be relevant only for certain baselines. For example, it’s no good trying diesel if you have a engine that runs on gasoline. But that doesn’t mean diesel is not relevant to the speed of the car. You have to try it when you have a diesel engine.
Another caveat is that you need to toggle factors on top of a non-default outcome. For example, the default outcome for a car is to remain stationary. Toggling or pressing the accelerator will work only when the ignition is on and the handbrake is released. Otherwise, it will seem useless. So, if the system needs several relevant factors and some of them are missing and you toggle a factor, you will not learn anything about that factor because the outcome will remain the default outcome.
Finally, you have to toggle just one factor at a time. Otherwise, you will get a large diff and will find it harder to figure out which factor caused the change. For example, if you press the accelerator and the brake at the same time, the car won’t move and you won’t know if the accelerator is just there to make extra noise along with the brake.
Examples
Example: Consider what Harry Potter did with his new mokeskin pouch in HPMOR. He was told: if there is some item in the pouch and you say the name of the item in English when your hand is near the mouth of the pouch, you will get that item in your hand.
What are the relevant factors you can see in that little model?
.
.
.
.
Answer: Pick out the verbs and nouns. Here are a few: some item in the pouch, you, say, name of the item, English, hand, near the mouth of the pouch.
How can you test if these are truly relevant factors? Toggle them.
The toggled factors become: some item not in the pouch (perhaps some item just outside it or not existing anywhere at all), not you (someone other than Harry), don’t say (perhaps mime or think or write), not the name of the item (perhaps a synonym or an alternate description), not English (Japanese or Pig Latin), not hand (foot or mouth or nothing at all), not near the mouth of the pouch (one foot away from it or ten feet away or ten miles away or under the pouch instead of near the mouth).
Harry had put in a bag of gold coins in the pouch. Here is what he did, along with my comments:
“Bag of element 79,” Harry said, and withdrew his hand, empty, from the mokeskin pouch.
[toggle the name of the item; use the technical name of gold - nope, didn’t work. The English name was relevant.]
…
“Bag of okane,” said Harry. The heavy bag of gold popped up into his hand.
[toggle the language; use the Japanese word for gold; worked! English language was not relevant at least for this baseline. That is, when you say “bag of X”, you can describe X in any language.]
Harry withdrew the bag, then plunged it again into the mokeskin pouch. He took out his hand, put it back in, and said, “Bag of tokens of economic exchange.” That time his hand came out empty.
[toggle the name of the item; describe the coins without the gold; failure could be due to the lack of mention of gold or the economic description; should try bag of gold tokens of economic exchange]
“Give me back the bag that I just put in.” Out came the bag of gold once more.
[toggle the name of the item; describe the bag and the time at which it was put in without mentioning the gold or the coins; worked! Seems like, when you mention the time of placement, the contents (gold coins) are not relevant. But they may be relevant when you don’t mention the time. Have to test.]
Harry James Potter-Evans-Verres had gotten his hands on at least one magical item. Why wait?
“Professor McGonagall,” Harry said to the bemused witch strolling beside him, “can you give me two words, one word for gold, and one word for something else that isn’t money, in a language that I wouldn’t know? But don’t tell me which is which.”
“Ahava and zahav,” said Professor McGonagall.
“That’s Hebrew, and the other word means love.”
“Thank you, Professor. Bag of ahava.” Empty.
“Bag of zahav.” And it popped up into his hand.
[toggle the language along with the factor of “do I know this language?”. Still worked! Not a relevant factor for this baseline. Have to also toggle “bag of” to see if that was relevant.]
“Zahav is gold?” Harry questioned, and Professor McGonagall nodded.
– Chapter 6, HPMOR (comments mine)
When I first read this, I couldn’t really figure out how he chose his tests. It felt pretty strange and arbitrary, like some arcane thing that only scientists could do. I hope you can now see how easy and non-mysterious it is.
Since he wanted to retrieve a bag of gold coins, he toggled factors like “gold” and “coin”. Can you think of a factor he didn’t toggle?
He could have tried “give me back what I just put in” instead of “give me back the bag I just put in”, so as to toggle the bag. Remember to toggle all kinds of factors, even ones that seem pretty relevant. Maybe they aren’t.
Note that he also toggled a factor that was not mentioned in my original, simplified description: “do I understand the language?” He could tell that the pouch must be using something to recognize natural language descriptions and suspected that it might be using his abilities. So, he toggled his abilities by using a word he didn’t know.
Other tests he could have done were: toggling his hand, i.e., not putting his hand near the pouch, perhaps by using his foot or mouth or nothing at all; toggling the distance of his hand from the pouch to see when it would stop putting the item in his hand; toggling whether the item is actually in the pouch or outside it (who knows, maybe the pouch can Accio stuff from your room); toggling whether the item exists at all, like a chocolate bar when you don’t have any.
Also, I suspect that the other contents in the pouch are relevant. If there is only one bag, maybe you can say “give me the bag”, but if there different bags, you have to say “give me the bag of chips”.
Later, when he’s unable to speak, Harry toggles the saying of the word out loud (just as our list suggested):
Harry put his hand into his pouch and tried to say ‘marker’ but of course no words came out. For one brief moment that stopped him; and then it occurred to Harry to spell out M-A-R-K-E-R using finger motions, which worked. P-A-D and he had a pad of paper.
In short, describe the model in as much detail as you can, list out its relevant factors, and toggle them to see if they really are relevant. If they aren’t, then you win because you no longer have to waste time providing unnecessary inputs and you can do stuff even when some factors are unavailable.
Example: I have a friend who’s pretty scared of ghost movies. I wanted to figure out what were the relevant factors in making a movie scary so that I could convince him to watch them with me sometimes. How would I describe an example? I could see that he and I got scared when watching the visuals and sounds of a ghost movie in a dark room.
How could I find out the relevant factors?
By toggling the factors. I muted the sound on the ghost movie and dared my friend to watch it with me for at least two minutes. He did so with some trepidation and found to his surprise that the movie wasn’t all that scary. Yes, the jump scare moments where the ghost pops up on the screen were still pretty thrilling, but the movie seemed to have lost its edge. I found out the sound was a major factor in making a movie scary. In fact, the movie was so tame without the sound that another friend complained that I was cheating. So, instead of muting it, I lowered the volume. It still wasn’t as scary as the original. Therefore, loud sound effects were definitely important.
Next, we tried “watching” the movie with the sound on but without the visuals. It was still pretty scary, if I remember correctly, but not as scary as watching it with visuals. So, the visuals were important but perhaps not as much as the sound effects. I could also have toggled factors like the darkness of the room, our distance from the screen, and the speed of the movie.
This trick of muting the sound also seemed to work on “addictive” movies and YouTube videos. They started to seem boring without the sounds and dialogues even though everything else was still the same: the fame of the entertainers, the clever editing, and the visual effects.
Example: Scientific experiments, of course, work by toggling factors. For example, spontaneous generation was a theory in the 19th century that held that “air everywhere could cause spontaneous generation of living organisms in liquids”. One proponent said that “the air, being everywhere the same, had the power, no matter where it was gathered, of causing the creation of [organisms]”.
One experiment that supported that claim was done by Louis Pasteur. He toggled the access of air to boiled liquid by stoppering a flask and showed that no organisms grew in the liquid. When there was no stopper, organisms had grown in the flask. That meant something about air was a relevant factor for organisms growing in the liquid.
But Pasteur didn’t just stop at “air is a relevant factor”. In fact, the whole controversy between him and the spontaneous generation theorists was that he said there were organisms already in the air that led to more organisms whereas they said that air somehow spontaneously generated organisms even though it didn’t have any organisms within it.
So, he unpacked the high-level factor of “air” into smaller factors. If you describe air in detail, you can see things like dust particles and so on. He needed to toggle the dust particles, i.e., get air without the dust particles. He did so by using a swan-necked flask that made the dust particles stick to the tube instead of reaching the liquid. Nothing grew in the liquid.
Isn’t this enough to show that dust particles are relevant factors? Air with dust particles led to organisms growing last time and air without dust particles led to none growing this time. Yes, but he toggled two factors from the previous positive example of organisms growing: the access of dust particles and the shape of the swan-necked flask. Maybe a swan-necked flask always prevents organisms from growing. To narrow it down to the dust particles, you have to toggle just the dust particles and get a change in the output. So, he tilted the flask to make the liquid touch the dust particles. Organisms grew in the liquid. Now, he had a one-factor diff and had successfully shown that the dust particles (or whatever it was that got trapped in the swan neck) were relevant for the organisms growing, and that the organisms weren’t spontaneously generated from just the liquid and the air.
Example: Writing - Consider the following piece of advice where William Strunk, Jr. toggles the active voice to passive voice:
Use the active voice.
The active voice is usually more direct and vigorous than the passive:
I shall always remember my first visit to Boston.
This is much better than
My first visit to Boston will always be remembered by me.
The latter sentence is less direct, less bold, and less concise. If the writer tries to make it more concise by omitting “by me,”
My first visit to Boston will always be remembered,
it becomes indefinite: is it the writer, or some person undisclosed, or the world at large, that will always remember this visit?
– The Elements of Style, William Strunk, Jr.
The subject, verb, and object remain the same; only the voice differs. It went from “I did X” to “X was done by me”. Yet, we can feel a difference in impact.
Another experiment is: “Great is Diana of the Ephesians” vs “Diana of the Ephesians is great.” The former sounds more poetic; the latter more matter-of-fact. All because of a shift in two words from the front to the back. You can similarly toggle factors like parallelism, verb placement, and the rule of threes.
This applies to techniques in general. To find out the impact of a technique, keep everything else constant and look at the result before and after applying the technique.
In fact, the best way to know if a technique is useful is to find a problem that you couldn’t solve five minutes ago and show that it can now be solved after learning about the technique. That’s a diff of size one that leads to a change in the output from unsolved problem to solved problem. You can tell that the technique was definitely a relevant factor. For example, I wasn’t able to figure out how Harry came up with those tests for the magic pouch (and for all kinds of problems through HPMOR). Once I hit upon the idea of toggling factors, it was a piece of cake. Toggling for the win!
comments powered by Disqus