The Black Box: Eyewitness Testimony and Investment Models (GestaltU)

by Adam Butler, GestaltU

We’ve been spending a lot of time recently discussing the quality of investment modeling, and the reliability of back-tests.  Specifically, we covered multiple discovery and degrees of freedom as two compelling reasons for out-of-sample performance decay.  Both of these sources of decay relate to the model itself.

Multiple discovery suggests that the most valuable, achievable advances in a field are often being examined simultaneously – yet independently – by many people at the same time.  It stands to reason that on these occasions, leaps in logic can often occur at the same time by independent parties.  And even in the cases where an individual makes the leap, it doesn’t take long for intelligent, competing parties to use reverse engineering to “catch up.”

Degrees of freedom relates to the counterintuitive notion that the more independent variables a model has – that is, the more complicated it is in terms of the number of independent ‘moving parts’  - the less reliable a back-test generally is.  This is because more independent variables create a larger number of potential model states, each of which needs to meet its own standard of statistical significance.  A model that integrates a great many variables seems like it would be robust; to the contrary, it is likely to be highly fragile.

Today, we endeavor to broaden the topic of degrees of freedom by adding a layer that is all-too-often ignored: your (our) behavior.  As quantitative investors and researchers, we generally don’t like to work in “squishier” areas of social science.  As trusted financial advisors, however, we know that when we sit across from clients, they often need more than just an evidence-based approach to investing.  Often, they need encouragement, nurturing and coaching.

People use facts as factors in decision making, but they take action on emotion. We engineer investment strategies that not only work in silico, but that also work in practice with real clients whose behaviours and actions are never completely removed from their emotional state.  In other words, the rules based approaches we apply in practice need to be compatible with the much less predictable black box inside your (and our) skull.

In the world of statistics, there are classifications for different types of variables.  We spend a great deal of time on this blog talking about “system variables.”  These are the rules which guide our research and investing.  They relate to how we examine and stress test models to achieve statistically significant results and how we ultimately make investment decisions.  These variables are procedural, and they are eminently controllable.

We’ve spent less time on this blog – at least recently – discussing “context variables.”  These variables relate to the investor specifically, and to their cognitive and behavioral responses to a given set of circumstances. For example, each investor has a slightly different reaction to gains and losses of different magnitudes. These are sometimes called “estimator variables” but we think “context variables” is a more intuitive term.

If the difference is unclear, imagine that you are brought in to a police station for the purposes of providing eyewitness testimony.  The police will have a procedure that they put you through.  Will you look at a lineup of real people?  Will you look at pictures?  If you do look at pictures, will they be sequential?  In sets of 6?  While you’re doing all of this, will an officer be looking over your shoulder?  What gender, race and age will the officer be relative to the witness?  Or relative to the suspect?

All of the variables in this process are “system variables” because they are under the direct control of the person managing the system.  It’s a choice to do an eyewitness identification using one procedure versus another.

Now imagine your specific mental state while sitting in the police station.  How might your mental and emotional status change depending on the nature of the crime?  What if there was a weapon involved – would you focus on the weapon or the assailant?  How confident would you be in your identification if the crime happened an hour ago versus a day ago versus a week ago? In your neighbourhood vs. a neighbourhood far from your home? Are you more or less likely to select a picture reflective of your own race or sex? Are people with tattoos miscreants or creative types?

All of these are “context variables” because they relate to you and the context surrounding your individual decision making process, in this case your ability to provide accurate eyewitness testimony.

To be clear, there is a relationship between system variables and context variables; they are not completely independent of each other.  For example, optimizing system variables by implementing procedures that decrease anxiety and decrease the amount of time between the crime and the identification can help stabilize otherwise volatile context variables, leading to more accurate eyewitness testimony.

Investing operates in much the same way.  We constantly endeavor to explore new investment methods, integrating ideas where appropriate and putting into production system variables that show strong statistical significance.  And we know that if we are successful, we will likely have a muting effect on otherwise volatile context variables.  In plain English, if we design a system that delivers stability and growth, we know our clients are likely to make more rational financial decisions and generally show higher levels of commitment to their long-term investment plan.

Unfortunately, this won’t always be the case, which brings us back to performance decay.  Every year, DALBAR releases their updated Quantitative Analysis of Investor Behavior (QAIB).  Predictably, it shows that the average investor does significantly worse than a simple buy-and-hold investor.  Much of this performance gap is attributed to behavioral deficiencies (aka context variables); a great many investors trapped by cycles of fear and greed buy high and sell low.

One issue that isn’t addressed by the QAIB is the notion that there exists a connection between system and context variables.  If you are investing in the S&P 500 where 6-month price volatility since 2000 has had zen-like lows near 7% and mania-inducing highs above 58%, it seems almost natural that your responses would follow a predictable downward spiral of doing the exact wrong thing at the exact wrong time.  In other words, the investment system isn’t completely blameless in the examination of emotionally flawed investing.

Volatile investment results induce volatile emotional responses, almost always to the investor’s detriment.  1987 notwithstanding, “buy and hold” was relatively easy to do from 1982-2000; it’s been an emotional roller coaster ever since.

We have a motto in our office: “We’d rather lose 50% of our clients near the peak of a runaway bull market, than 50% of our clients’ assets during the inevitable bear markets.”  If most Advisors are honest with themselves, they will admit that their advice leads to precisely the opposite outcome.  To wit, an Advisor advocating “Strategic Asset Allocation” – or a “buy and hold” philosophy - with a large equity component is definitionally acting in a way that is inconsistent with our philosophy. That’s because this type of portfolio can expect a 30% – 50% loss in value about once every 7 years.  This Advisor will collect most of his clients near the end of a long bull market when his near-term performance has necessarily been strong. Soon after these clients will endure a major loss. This is a nearly universal cycle in wealth management.

…hence, why our motto stands out.

In our Adaptive Asset Allocation method, we’ve endeavored to deliver impressive results while focusing intensely on risk controls.  Because of this, we know that there will be times when our model underperforms whatever stock index is in the headlines.  We know that sometimes this  underperformance will endure for extended periods of time.  Further, we know we will almost certainly lose some clients near the end of this bull run. It’s happened before, and it will happen again.

But the difference is that we find it impossible to look our clients in the face when the stock market is down 50% and say that we succeeded by only losing 45%.  That’s in our DNA. And it’s why we encourage investors to analyze the performance of any Advisor under consideration over an entire market cycle, which includes both bull and bear markets. In doing so, we also believe that we’re helping our clients “short circuit” the vicious cycle that the QAIB annually revisits.

After all, should we judge sports teams only on how they perform in the first half of the game? Or does the back half matter?

It’s true that we don’t spend as much time on this blog as our colleagues might discussing behavioral finance.  Now you know why: the best way we know to limit the adverse effects of such behaviors is to provide our clients with a return profile that doesn’t compel them to make bad choices under duress.

Copyright © GestaltU

The post The Black Box: Eyewitness Testimony and Investment Models appeared first on GestaltU, March 7, 2014.