How To Radically Improve IT Decision Making
Cognitive biases and limited alternatives can greatly reduce business and especially IT decision quality. In this podcast, Matthew O'Keefe, VP at Oracle, and Ian Corner, a leading IT architect in Australia review cognitive biases that impair decision making and how to counter them. In addition, we show how you can cheaply add low-risk alternatives to your decision palette to maximize innovation, efficiency, and time-to-market.
We reference this paper many times in this podcast:
Before you Make that Big Decision by Kahneman et al
https://hbr.org/2013/06/before-you-ma...
00:00 Matthew sets the stage for the discussion: why biases and low-cost alternative exploration are critical to good decisions.
04:00 We describe evolutionary prototyping and how it allows you to quickly and at low cost iterate on design ideas, assessing risks, costs and opportunities.
06:00 Evolving a file system design via evolutionary prototyping.
07:15 Ian Corner joins the podcast, introductions, etc.
11:00 More on evolving system designs iteratively.
12:30 Importance of domain knowledge, time bounding, each subcomponent does not have to be perfect.
14:00 Steve McConnell's "Rapid Development" is a fine resource for best practices in design and software engineering.
15:00. Organizational issues around considering multiple design alternatives.
18:00 Things that make design hard, including long term survivability, performance, discoverability, etc. Hard, expensive complex decisions versus simple, quick decisions based on intution.
19:40 Introduction of human biases, Kahneman's work that won the Nobel Prize in Economics, including fast (System 1) and slow (System 2) decision making.
22:00 Self-interested bias in IT examples, including technical workers favoring "hot" technologies they can put on their resume.
30:00 Organizational cultures and examples of considering multiple alternatives.
31:37.Affect heuristic, when staff fall in love with an idea and won't consider alternatives.
33:30 Why Ian hates all technology and why not falling in love with them helps you make better decisions to achieve specific outcomes.
36:00 Why you should your workflows and processes first without technology.
36:45 Lack of dissenting opinions is a red flag. Ian points out in Australia it's less of a problem, culture supports lots of discussion and debate. Lack of debate can happen due to pressure from senior leadership to focus on one approach that may not be the best.
40:15 Saliency bias occurs when a decision maker assume a different, successful event must be related to the the problem at hand, when it really isn't. Ian points out you always need to go back to your business outcome and what are you reaching for. How to get there comes after. And often you may have multiple business outcomes you have to achieve, and one solution cannot do that. Ian's example here is desktop visualization requirements for his workforce.
45:00 Cognitive biases cause us to filter information to fit our perceptions. Same process can be applied to any technical decision aiming for a specific business outcome.
47:00 Standards help make it easier to make good decisions when they lead to multiple implementations with differing characteristics.
49:00 Biases based on limited information, when future information would really help you make better decisions. The way to counter this is to find ways to get the information sooner in the design cycle. Evolutionary prototyping and designing for extremes can help here.
52:00 How to design for scale. Plan to thrown one away.
57:00 Halo effect. Companies, people, and products that succeed get a halo associated with their approach so that people are biases to believe their approach or product can extend beyond the domain in which it was effective. How Matthew helped Ian understand that US supercomputing wasn't succeeding in quite the way he thought, he was associating a halo with the US supercomputing community's techniques.
59:30 How IT cargo cults destroy effective decision making. Why cargo cults develop in general and IT culture in particular.
1:05:00 How to think upstream about what will happen downstream if you don't normalize your data.
1:11:30 How to introduce risk-based thinking into your decision making, and how to help management become comfortable with this. Waiting to deal with a risk only when it happens is dangerous, instead retire them or find tradeoffs that reduce them.
1:13:30 You can't use agile or sprints ALONE to get to a great design. Agile has many advantages but it is not sufficient to reach great design. Sprints work great for prototyping.
1:15:30 Why Seymour Cray tried to innovate in only one design dimensions at a time. Long term integrated designs take a long time to get right.
1:17:00 How to roll out infrastructure in an agile manner using modularization.
1:18:30 Optimistic bias and its effects. Baseline optimism is important.
1:21:00 Improving culture.