Date: 2023
Machine Teaching: What I Learned From My Optimizer - The D. E. Shaw Group
- In finance, optimizers are used by a range of market participants to help solve different types of problems, informing asset allocation decisions made by institutional investors and risk allocation decisions within banks and insurance companies
- In these discretionary contexts, we’ve found that beyond helping humans make decisions about the portfolio at hand, optimizers through their output can also teach humans—including how to better design and interact with an optimizer itself, and, importantly, how to counter behavioural traps and cognitive biases when making investment decisions.
- While the science of optimization is heavily grounded in math, the art of optimization—improving design, inputs, and hopefully outputs—comes from confronting it with human intuition, and vice versa. It’s this iterative process that provides learning opportunities for the human practitioner.
- Why does the optimizer’s desired trade size differ so meaningfully from what I expected? Did I miss something fundamental when estimating my parameters? If not, what is the optimizer’s output telling me about my own assumptions, judgment, and possible biases?
- It’s tempting as investors to try to constrain portfolios with surgical precision. An obedient optimizer reminds us that it’s not so simple in the real world, where every objective comes with trade-offs. If a constraint is too ambitious or inflexible—or the conditions unfavourable—it can lead to unexpected costs. We’re encouraged to keep top of mind the law of unintended consequences when thinking about how to design robust targets and constraints for a given portfolio.
- As we’ve seen so far in this piece, and will see again, the output of an optimizer is highly sensitive to its inputs.
- Many traders are disposed to think about—and care a lot about—the daily risk of their individual positions. That may be a function of certain human tendencies (e.g., risk and loss aversion) and of institutional constraints (e.g., risk targets or limits). That narrow focus, however, can obscure the fact that it’s the aggregate utility of a portfolio over time that matters, which in turn is a function of the lifetime risk and return of the individual trades it comprises.
- A well-designed optimizer can help counter those shortcomings and reinforce the importance of correlations, the utility of diversifying assets, and the disutility of assets that generate unwanted or overlapping exposures.
- More generally, as we’ve seen elsewhere, an optimizer can help an investor widen their lens from the narrow properties of a single position to those of an entire portfolio, where an individual trade’s utility can only properly be understood in relation to all others.
- Imagine a trader has two forecasts, one an idiosyncratic forecast on silver futures, and one a forecast on lean hogs. Assume that the two forecasts are alike in all ways except that the lean hogs trade currently finds strong consensus among speculative investors. The trader’s base inclination might be to size the lean hogs trade larger in the portfolio, given the comfort they find knowing that other investors share their forecast. This intuition isn’t necessarily wrong, but it is incomplete. Markets are generally efficient, and there can be useful information in what other participants believe. At the same time, that consensus may indicate the presence of common investor risk.
- What an optimizer can consider more readily is the entirety of the portfolio in question. Across that portfolio, there are likely many individual trades, some of which have their own exposure to common investor risk. An optimizer can effectively manage an overall budget for common investor risk. Beyond a certain level, the bar for incurring one marginal unit of such risk can be high.
- Because of its ability to factor in low-probability but high-magnitude outcomes at the aggregate portfolio level, an optimizer can help investors appreciate how serious common investor risk can be and how to avoid assessing it too locally.