If you remember back to your math classes in college, it turns out that setting yourself up to create blends of ideas is particularly important for innovation – in terms of finding mathematically optimal solutions to complex problems. Sound crazy? Let me start at the beginning.
Our intuition as software engineers often leads us to believe that if we find a halfway-decent working solution to a problem, all we need to do is continually iterate and improve on that initial solution to eventually reach the best, most-optimal solution. This is one area where the Agile philosophy of choosing the simplest possible solution for any problem can run you aground unintentionally. Yes, it is still a reasonable heuristic that the simplest solution is often the best place to start, and we certainly don’t want to over-engineer for hypothetical future flexibility or architectural elegance. Sometimes, however, the simplest technical solution creates a suboptimal user experience or carries other hidden tradeoffs, and you may not realize those downsides until much later if you aren’t looking broadly enough. Your instincts often lead you to imagine the problem space to look something like what’s shown in Figure 1.
You imagine that all possible solutions are on the path to the one, optimal solution. You figure that once you get going on building something, you’re somewhere on the foothills of that mountain peak, and as long as you keep iterating and improving your solution, eventually you’ll reach the top—that is, you’ll land on a globally optimal solution. Engineers are natural iterators who constantly seek to improve their solutions by fixing bugs, adding functionality, reviewing designs with the team, integrating more components, improving error handling, and so on. So, by and large, engineers do climb whatever hill they’re on and make improvements little by little.
However, most engineers typically iterate only one solution idea at a time, and that’s where the trouble starts. The reason that iterating only one solution at a time is troublesome is that most problems are complex and multifaceted, and you’re trying to solve for many variables at once. The landscape of possible solutions is much more complex than you often realize at first. There are actually many good solutions to any given problem, perhaps even more than one great solution, and those solutions may be quite different from one another both in concept and in implementation. Each potential solution may trade off key aspects of implementation elegance, performance, scalability, usefulness, usability, and desirability in dramatically different ways.
Consider a much more complex, three-dimensional mathematical plane, Figure 2, where the good solutions to a given scenario are represented by local maximums—small hills on the contour diagram. There are also a handful of truly great solutions that have much higher peaks, and one outstanding approach that is the clear global maximum. If you start your project with a reasonably good, workable approach in mind, you can gradually improve your solution as you iterate and will likely climb to reach one of those small peaks, or local maxima. If you are very lucky, you might happen to start off on the foothills of that highest peak and reach the global maximum—because you picked a good starting point, or, in mathematical terms, a good seed.
But what if you are not so lucky? What if your first good idea is destined to lead you to a suboptimal solution, no matter how much you iterate it? There’s got to be some way to give yourself the best odds at iterating toward the most globally optimal solution and not getting stuck at the top of a much smaller hill. Rather than enthusiastically rushing to code up the first good approach you come up with, what if you start off by exploring and iterating several diverse approaches, trying out ideas that represent several different regions of that complex solution plane? What if you combine and recombine the best aspects of those ideas to help you identify which neighborhood of solutions is most globally promising and provides the best balance of tradeoffs for your situation?
This is actually a core idea behind finding global maximums (or global minimums) in mathematics, a problem that comes up commonly in the natural sciences. If you want to find the absolute minimum energy configuration of a particular chemical compound, it turns out that is a very, very hard math problem to solve. In fact, there is no algorithm that works for any arbitrary situation. There are several different algorithms used for different situations, but they all share a common strategy. The way to find an optimal solution fundamentally goes like this: start with many different possible points, or “seeds,” across a complex landscape, calculate the results for each seed, compare the results, and use that to gradually narrow in on the global optimum. Later, rinse, repeat, until you can’t climb any higher.
Even for mathematicians and physicists, no matter which algorithm they use, finding a true global optimum is a genuinely difficult endeavor. Thankfully, even though the software problems we work on are complex, finding the single, absolute maximum may not be as critical for us. In fact, there may be several comparable maximums, and practically speaking, we’d be pretty happy if we found any one of those truly great solutions and successfully avoided getting stuck on a smaller hill. To more reliably find a great, optimal (or nearly optimal) solution, you can take a lesson from the experts. To find optimal solutions in complex spaces, start out by casting a wide net of ideas and iterate multiple seeds as you narrow in on an optimal solution. As in mathematics, the more diverse a set of seeds you start your iteration with, the better the odds you have for discovering a more globally optimal solution in a complex universe.
It’s so easy to barrel down the wrong path, quickly picking an approach that looks reasonable, coding it up, and releasing it, only to find yourself fixing bugs and responding to customer feedback and pushing updates constantly, working hard but never realizing the success that seemed imminent with such an auspicious start. Later, when a competitor comes out with a variant that ends up gaining the popularity you hoped for, you may realize that the root problem was something more fundamental in your approach—the underlying architecture that defined the core performance characteristics, the user experience that needed to have been simpler and more streamlined, a key aspect of the customer’s end-to-end experience that you didn’t notice, the developer whose apps ended up driving customer adoption and purchase, or the worst one of all – that you did a great job solving a problem no one really cared about.
Looking back, the salient postmortem question often is “Did you ever consider any alternative approaches that might have led to something similar to your competitor’s winning solution?” Sometimes the answer is yes—that an intentional decision was made not to follow that path, and that will happen from time to time. But all too often the answer is no—that approach never occurred to you—you went down the first viable path that seemed to meet the constraints and didn’t even notice that there were other alternatives to consider.
Without a broad enough set of inputs, no amount of iteration will lead you there. That sort of failure lies not in the iterative process, but in not considering a broad and diverse enough set of alternatives from the very start.
Want to learn how to bring this mindset to your team? We specialize in bringing customer-focused innovation techniques to highly technical engineering teams.