Why I should study existing sources first

Question: Where did I think I had original ideas only to discover that other people had done it before and done it way better?

Blind Spot

This is an empirical study designed to smash my own intellectual arrogance and teach me the Virtue of Scholarship.

I want to show myself the places where I discovered others doing something much better than I did.

Why?

You have a blind spot around strategies that involve doing nice things for other people, to the point where it stops you from achieving your selfish values.

– Harry James Potter-Evans-Verres, Chapter 108, HPMOR

I have a blind spot around strategies that involve learning things from other people (instead of “inventing” them myself), to the point where it stops me from achieving my own values.

I try to “figure things out for myself” and thus fall heavily for the Not Invented Here Syndrome. I discover some ideas on my own and, because of Commitment and Consistency, I think they’re epic and awesome. Now, since they’re so awesome, I completely fail to study the literature on how people have solved the same problem. In other words, I fall for the Technique Trap.

Why so blind, son?

It’s not like I hadn’t heard of people who did a lot of literature research before writing their posts. Lukeprog on LessWrong is definitely a person like that. And I had read his explicit warnings a long time ago in The Neglected Virtue of Scholarship:

The lesson I take from these and a hundred other examples is to employ the rationality virtue of scholarship. Stand on the shoulders of giants. We don’t each need to cut our own path into a subject right from the point of near-total ignorance. That’s silly. Just catch the bus on the road of knowledge paved by hundreds of diligent workers before you, and get off somewhere near where the road finally fades into fresh jungle. Study enough to have a view of the current state of the debate so you don’t waste your time on paths that have already dead-ended, or on arguments that have already been refuted. Catch up before you speak up.

This is why, in more than 1000 posts on my own blog, I’ve said almost nothing that is original. Most of my posts instead summarize what other experts have said, in an effort to bring myself and my readers up to the level of the current debate on a subject before we try to make new contributions to it.

Then why didn’t I heed his words?

Well, every time I came across that passage, I thought Luke was being stupid - “in more than 1000 posts… nothing that is original”?! That’s crazy! When will you get started on, you know, actually solving some problems? You can’t wait for others to solve all your problems.

True. But, I now realize, you can’t not wait for other scientists to thoroughly test your favoured beliefs. You need to be super-empirical. One wrong step can take you anywhere. An Affective Death Spiral around a wrong solution is the default outcome. Unless you are extremely rigorous, that is what will happen. Our minds are that bad at thinking correctly.

Why didn’t I understand this point earlier?

I underestimate other people’s abilities, I think, and overestimate my own. They call this the Dunning-Kruger Effect, I believe:

The Dunning-Kruger effect is a cognitive bias wherein relatively unskilled individuals suffer from illusory superiority, mistakenly assessing their ability to be much higher than is accurate.

Yup, that’s me. Unskilled person with illusory superiority, especially in areas where I don’t get strong feedback about my own skill level.

I naively believe that other people aren’t very smart (unlike me, in my mind’s view). So, they must not have really achieved too much. I mean, if they aren’t really too powerful, they can’t have achieved much, right? And to support this illusion, I ask myself why they haven’t got great results, if they really are so skilled. (Note how I completely fail to apply this very high standard to myself.)

No Shame in Learning Existing Ideas

Why don’t I learn from others by reading and implementing techniques from as many books as possible? Why do I have this blind spot? It’s not just that I think I’m clever and others really aren’t. Beyond that ridiculous belief, I hold another one: you are a worthy thinker only if you discover important ideas on your own.

For example, I feel compelled to discover important programming ideas by myself. I did that for the Y-combinator function, sitting by myself for a couple of hours till I got the answer. It felt good, make no mistake. But it’s just not scalable.

Your problems don’t care where you get the solution from. All that matters is that you solve them.

So, don’t waste your time trying to derive the answer to every theorem in your math book by yourself (yes, I was stupid enough to hope to do that). That’s never going to happen. It probably took several years between the discovery of each theorem - check the Discussion section of the chapter to find out. That’s how hard we find math.

There’s no shame in going through all the math problems and solutions, analyzing them to figure out how the authors got from the problem to the solution, and then deploying those techniques on the problems you come across. All that matters is that you reduce the time taken for you to solve future problems. Yes, you should of course try to derive those theorems after closing the book to make sure you really have got it. That’s what it takes to load ideas into your memory. But don’t try to produce all your knowledge from some basic seeds.

Hmm… I seem to have this ideal of an all-powerful superintelligence that needs just a few seeds of knowledge from which it can then derive all human knowledge so far. That’s possible, sure, but I just can’t afford that much time. Given the raw data from all the experiments so far - all the X-rays ever taken, looked through, the questionnaires answered - you can definitely process the information and come to the most likely hypotheses. But it would take a supercomputer to do that.

Swallow your pride and just learn the final answers from the experts who have achieved at levels far beyond yours. For example, look at how adroitly Richard Bird solved the Sudoku problem. I couldn’t believe how succinct his boxes function was for getting the 9 boxes from a Sudoku matrix.

boxs :: Matrix a -> Matrix a
boxs = map ungroup . ungroup . map cols . group3 . map group3

I would never have come up with that, not unless I’d sat and thought about such problems for days and weeks on end. I was sure that you had to use index-based code to solve a Sudoku puzzle. Now, I could bumble my way through dozens of failed attempts and blow off all the other projects in my life before I reach one with the diamond clarity of the one above or… you know, I could just read his excellent book Thinking Functionally with Haskell and gain all of his decades of wisdom in a few hours!

I’ve been a fool.

Lesson: You can’t derive all the answers yourself. Learn the answers from the experts instead.

No More

Enough. No more stupid damn mistakes like these. Let me list out the earlier evidence so that I can see just how wrong I have been and so that I notice in the future when I have that curious feeling of illusory superiority.

Spaced Repetition

In around 2012, I had thought of the idea of a bunch of review questions that would cover all the important points in a textbook chapter. They were like the unit tests that cover all the important logic in a program.

And then I came across the idea of Spaced Repetition. The people using it were basing it on well-understood psychological facts (the Spacing Effect and the Testing Effect) and had automated it.

Any Spaced Repetition system worked better out of the box than my own vague, poorly-specified idea of “utils” (which is what I called my pseudo unit tests). If I had only thought to search around for testing systems already being used.

Notes on Website vs in Org mode

The reasons behind this are a bit complex. I’ll go into them in a separate essay. But, the point is Gwern got it right - have all your thoughts on your website. I got it wrong - org notes aren’t the best format for ideas that you want to evolve into other things.

The point is, I didn’t realize it. I thought Org mode was obviously the superior solution for storing your thoughts. The website is only for finished essays, I thought, not for fledgling ideas.

PG got it right, too. I can’t recall seeing even one list in all his essays. It’s all prose.

Shame and Perfectionism

I was thinking in terms of how I wanted perfection or nothing. But Brene Brown just blew me away with her masterful grasp on how shame, perfectionism, worthiness, and vulnerability are linked together. This is yet another place where I would have done well to consult the literature first instead of creating ad hoc theories of my own.

Rapid Prototyping

I thought I had understood the concept of Tightening the Loop very well and that I had some novel ideas about it. I believed that nobody else had realized the importance of “Tightening the Loop”.

Wrong! Turned out others had understood it far better than I ever had. Paul Graham had known it deeply for at least two decades and preached it far and wide. Startup founders generally know this idea like the back of their hands. The guys who wrote Continuous Delivery understand it spectacularly well - I sat through their entire first chapter spellbound.

In fact, this is when I started to generalize the notion that I was deluding myself about my special “understanding” of ideas. Open your eyes, I told myself. A bit less self-absorption, a lot more empirical scholarship. I know pretty much nothing right now.

Planning

I once thought up an idea called the Plan and Implement Fallacy. I thought I had hit upon something new and important that nobody else understood (I hope you can sense a pattern here).

Writing simply

I was a big fan of fancy text decorations - bold, italics, Capitalized Words, lists, quotes, corollaries, etc.

I didn’t find PG using any of them, in his hundreds of essays. Now, I understand why.

Empiricism

I was trying to invent my own methods of how to be empirical. But, scientists have been writing journal papers empirically for a long time now. They back every statement with citations showing where they got the evidence from. They are masters of the art. I need to catch up to the state of the art first before trying to create my own “techniques”. I need to analyze their research papers and learn where and how they decided to add citations. They probably have tutorials and such that teach newbie scientists. I need to grab hold of those.

Causality

I was struggling to formalize the notion of causation that scientists seemed to use. I found it hard to accept the informal scientific method because I couldn’t see how to explain it to a computer. Plus, it didn’t jibe with Bayesian inference, which I had heard was the basic rule behind correct thinking. Bayesian theory was all about probabilities and such, whereas scientific models seemed to make forthright predictions. Then I found that computer scientists like Pearl had already figured it out. As he himself says: “Put simply, causality has been mathematized.”

Categories

I was trying to figure out how humans reduced the space of possible hypotheses over atoms to a space that only contained high-level variables like GDP or car or friendship. After a long struggle, I found that Eliezer had essentially answered the question of “where do variables come from?” in his sequence on words and categories.

Haskell Techniques

People have already figured out how to use functors, applicatives, monads, monoids, lenses, etc. to abstract their code to ridiculous levels. You need to get to their level of skill before you start trying to “invent” some “epic awesome” technique.