Measuring Performance: A New Direction

Measuring Performance: A New Direction
Fiduciaries often use flawed methodologies to evaluate manager performance. There is a better way.
by Bruce Curwood

Current investment practices are all too often an outgrowth of internal politics and history, steeped in bureaucracy and resistant to change. Unwieldy committees of well-intended fiduciaries attempt to deal with rapidly changing markets by evaluating performance, with flawed methodologies, over too short a time horizon. Failing to realize their targets of first quartile performance and frustrated with disappointing results, funds quickly dispense with their current manager complement, in many cases seeking out last year's fad.

Clearly a new process is needed. The key is fiduciary communication. The myth of comparative analysis (universe comparison) must be stripped from the fiduciaries' repertoire and the limitations of quantitative analysis exposed, so that fiduciaries firmly embrace the precept of qualitative analysis, ratified by quantitative performance. Although greater resources, time and tools are necessary, the end result is clearly superior decision making, with lower pension impact costs.


Current Practice
Before putting forth a solution, let's begin with an analysis of current practice.To evaluate individual manager performance, most Canadian fund sponsors use a combination of quantitative analysis and intuition, over a four-year time horizon (the stated norm for evaluating plan and manager performance, according to the typical Canadian Pension Statement of Investment Policies and Procedures. In practice, however, most investment committees' pain threshold commences at about two years and action is generally taken well before the stated four-year evaluation period. But let's not quibble over time horizon; as we'll see, even four years is too short).

Why do fund sponsors use this approach? Quite frankly, it is the accepted norm, readily available, easy to use, naively appealing, and relatively inexpensive.

Everybody likes the idea of watching a performance horse race and witnessing the definitive winner emerge, based on a finite quantitative test. After all, it's far less complex than analyzing a multitude of qualitative and quantitative variables and painstakingly trying to interpret the best solution for your fund.

But there are two main flaws with the current evaluation process:
* Quantitative analysis is not predictive and is often inaccurate; and
* Group intuition is subject to several behavioural impediments and is not objective.


Quantitiative Analysis
One of the most common tools used in quantitative analysis is universe median comparisons, or relative performance measurement. Unfortunately universe median comparisons are poor benchmarks, as they are ambiguous, not specified in advance, not investable (can't be replicated), not reflective of current investment opinion, and inappropriate due to varying styles, asset mix and plan risk tolerance.1 In addition, they are subject to survivorship bias.2

The rest of the quantitative tools (benchmarks, scattergrams, tracking error, value added charts, information ratios and attribution analysis) are at least helpful, but are not predictive or consistent,3 and are unable to discern luck from skill without a great deal of data and time, both of which are often unavailable.

The latter point is particularly relevant, given the four-year time horizon generally used to assess managers. Is there any magic to four years? No. The concept probably evolved from the four-year presidential cycle, which most people assumed coincided with the business cycle and, therefore, capital market returns.

According to Statistics Canada, there have been six full Canadian business cycles between 1960 and 1998. The average business cycle lasted about 6.3 years, while the longest was almost nine years. In fact, the last two have each been over eight years. Only the shortest, at 3.75 years, is close to the evaluation period used by most investment plan sponsors.4

Given that four years is not a useful time frame, how long does it take to discern luck from skill on a pure quantitative basis?

The chart below shows the required time horizons, by asset class.

With a minimum of at least 16 years required to statistically determine if managers can add value due to skill, we see that what plan sponsors are measuring over a shorter time period is just white noise. This demonstrates the futility of using quantitative analysis over a four-year time horizon to evaluate manager performance. Is it any wonder that, on average, pension funds fail to add value to their passive composite benchmarks?5


Group Intuition, A Recipe for Disaster
The second major flaw associated with current investment plan practices is the group dynamics associated with investment committee decisions. The following observations reflect my personal experience, but are not meant to be all-encompassing, nor do they necessarily apply to all fiduciaries.

Firstly, high-level fiduciaries are well intentioned, but they usually have busy, full-time careers. Their services on behalf of the pension or endowment plan are often on a volunteer basis. Although in the majority of cases they perform their fiduciary functions in a conscientious manner, they are often time-sensitive and can be impatient to return to their pressing full-time endeavours.

Secondly, fiduciaries have varied backgrounds and in many cases are experts in their own field. While this is helpful in bringing different perspectives to the investment committee table, few have pension expertise and most reveal their bias for business and corporate profit. Issues such as time horizon are considerably different for pension and endowment funds, which are long-term in nature, and business enterprise, where four quarters is an eternity. Used to taking charge and making a short-term impact in the corporate sector, these captains of industry want simple answers to what in many cases are complex investment issues.

Another trait is their suspicion of the new and unknown. They constantly look for problems or the downside, which is only natural, given the potential for personal liability. However, this may impede fund progress. Success in many instances belongs to those who conduct good research, do proper due diligence and implement on a timely basis. Investment funds need to be one step ahead of mercurial markets and to keep pace with new investment vehicles such as derivatives, asset-backed securities and inflation-linked securities.

Finally, fiduciary turnover may be an issue for many investment committees. New members, who did not take part in the initial board in the existing policy and strategic decisions, must be shown their merits and properly indoctrinated. A good paper trail, such as a statement of investment beliefs, reviewed with the Chair and Sponsor, as well as formal education, would surely assist in this indoctrination process.