I first came across novelty search in Farnam Street’s Slack. One of the fellow members had posted this talk and I thought that this was just one of the most interesting things that I’ve seen. I picked up the book “Why Greatness Cannot be Planned” and I’ve posted my notes below.
Experiments in evolutionary artificial intelligence demonstrate that progress toward an important, difficult goal is not best achieved by attempting to go directly toward that goal, but rather, by rewarding novelty.
Objectives might sometimes provide meaning or direction, but they also limit our freedom and become straitjackets around our desire to explore. After all, when everything we do is measured against its contribution to achieving one objective or another, it robs us of the chance for playful discovery.
In a sense we never stop playing the hot and cold game. The process of setting an objective, attempting to achieve it, and measuring progress along the way has become the primary route to achievement in our culture.
One of the reasons that objectives aren’t often questioned is that they work perfectly well for more modest pursuits.
It’s useful to think of achievement as a process of discovery. We can think of painting a masterpiece as essentially discovering it within the set of all possible images. It’s as if we are searching through all the possibilities for the one we want, which we call our objective.
In other words (and here is the paradox), the greatest achievements become less likely when they are made objectives. Not only that, but this paradox leads to a very strange conclusion—if the paradox is really true then the best way to achieve greatness, the truest path to “blue sky” discovery or to fulfill boundless ambition, is to have no objective at all.
The key problem is that the stepping stones that lead to ambitious objectives tend to be pretty strange. That is, they probably aren’t what you would predict if you were thinking only of your objective.
It often turns out that the measure of success—which tells us whether we are moving in the right direction—is deceptive because it’s blind to the true stepping stones that must be crossed.
We want to show you that it’s possible to explore a search space intelligently even without an objective.
Sometimes the best way to achieve something great is to stop trying to achieve a particular great thing. In other words, greatness is possible if you are willing to stop demanding what that greatness should be.
The key is to be open to change, to a shifting landscape where appearances can be deceiving yet liberating at the same time. The great achievers are willing to abandon their original objectives and spring for opportunity when it arises. What is important in these scenarios is to avoid locking into rigid commitment to the original ambitious objective, and instead remaining mindful and open to where the present stepping stone might lead.
If you take one thing from this chapter, perhaps it should be that you have the right to follow your passions. Even if they deviate from your original plans or conflict with your initial objective, the courage to change course is sometimes rewarded handsomely. Another important implication is that not everything in life requires an objective justification.
To arrive somewhere remarkable we must be willing to hold many paths open without knowing where they might lead.
But the truth is that novelty is no less information-rich than the concept of the objective. It’s just different information.
Rather than relying on a false compass, novelty only asks us to compare where we are with where we’ve been.
Deviating from the past is simpler and richer with information because we can look at the whole history of past discoveries to inform our judgment of current novelty. So it’s not unreasonable to believe that novelty is a meaningful engine for progress.
Behind any serendipitous discovery there’s nearly always an open-minded thinker with a strong gut feeling for what plan will yield the most interesting results.
In fact, this is an important insight: Because eventually you have to acquire some kind of knowledge to continue to produce novelty, it means that novelty search is a kind of information accumulator about the world in which it takes place. The longer the search progresses, the more information about the world it ends up accumulating.
Maybe the whole idea that there is some kind of “best practice” for finding your objective is misguided. There may simply be futility at the heart of search—no approach to searching can guarantee that you reach your objectives.
We can reliably find something amazing. We just can’t say what that something is!
The insight is that great discoveries are possible if they’re left undefined.
Of course, the story is never simple—being aimless isn’t always a good idea, but when it’s paired with a thirst for exploration, it might indeed hint at great potential.
Perhaps the effort invested in assessment would be better spent on trying different ideas without so much emphasis on accurate measurement.
Like in Picbreeder, some of the resulting ideas might be destined for failure and others for valuable discoveries, but the system (in this case society as a whole) benefits from all the divergent paths taken at the same time. Those approaches that look interesting or promising then become stepping stones from which other efforts can depart.
Through this process society becomes a treasure hunter for teaching methods.
For example, in Picbreeder, the reason so many images have been discovered is that users don’t agree on which images are better. Picbreeder works as a stepping stone collector because each user can follow his or her own path, even when others disagree with that path or wouldn’t take it themselves.
Seeking consensus prevents traveling down interesting stepping stones because people don’t agree on what the most interesting stepping stones are. And resolving this kind of disagreement often leads to a compromise between opposing stepping stones. Like a faded gray that results from mixing the sharp contrasts of black and white together, the product of such compromise often dilutes the two original ideals.
Even though someone else might later jump off from where a previous user left behind, never in the process is any consensus struck—and never is it needed to make discoveries.
Sometimes the best path to creative ideas is to explore as individuals, without consensus or objectives. Later others can extend the ideas of their predecessors, keeping the chain going.
Rather, the point is that disunity among research groups and within science as a whole can actually drive progress.
The point is, in the business world, it’s often also a good idea to follow the interesting, but when the idea is presented to investors, it needs to be clearly reachable from the current stepping stone. So the innovative solution must be discovered before approaching the investor.
So while betting on revolution may be dangerous, over time it does happen. But as with all great processes of discovery, the revolutions are rarely the objectives of the stepping stones that lead to them.