The number you were handed was the best of N
A Sharpe with no trial count is a numerator with no denominator. Here is what the record says happens when it gets priced anyway, and why the count has to ship with the result.
· 7 min · trial count
1 / 4
Somebody walks into a meeting with a Sharpe of 1.42. The curve behind it is clean. The story is plausible. What nobody in the room knows, and what almost nobody asks, is how many other curves were drawn before this one was chosen to be printed. Ten. Forty eight. Four hundred. The answer changes what 1.42 is worth, and the answer is not on the slide.
That missing number is the trial count. This note is about why it has to travel with the result.
Ten random tries give you 1.57
Start with the cleanest result in the literature. Bailey, Borwein, Lopez de Prado and Zhu, in the Notices of the American Mathematical Society in 2014, worked out what happens when a researcher tries a handful of configurations on data with no signal in it. With only 10 random configurations, the expected best in sample Sharpe is 1.57. Every one of the ten has an expected out of sample Sharpe of zero. The 1.57 isn’t a small bias to be trimmed. It’s the whole number.
They went further. With five years of data, no more than 45 independent configurations should be tried before the best in sample result stops meaning anything. Most sweeps run more than that before lunch.
The sentence from that paper on our Home page reads: “A researcher that does not report the number of trials N used to identify the selected backtest configuration makes it impossible to assess the risk of overfitting.” Not difficult. Impossible.
| trials | in sample | out of sample or haircut |
|---|---|---|
| 10trials | best in sample1.57 | out of sample0.00 |
| 48trials | 0.81walk forward | 0.63haircut |
| 72published strategies | median in sample0.85 | post publication55percent of that |
What the published record shows
You don’t have to take the simulation’s word for it. The record of published predictors has been measured after the fact, several ways, and the numbers rhyme.
McLean and Pontiff, in the Journal of Finance in 2016, tracked 97 published stock return predictors past the end of their original samples. Portfolio returns were 26 percent lower out of sample and 58 percent lower after publication. About 32 percent of the post publication decline they attribute to trading by people who read the paper. The rest is closer to this note’s subject: the published number was the best of a search, and the search was not reported.
Harvey, Liu and Zhu, in the Review of Financial Studies the same year, catalogued 316 factors published since 1967 and drew the obvious conclusion. Because hundreds of tests have already been run against the same history, a new factor should clear a t statistic of 3.0, not the 2.0 a single test would justify. The hurdle rises with the count, whether or not anyone wrote the count down.
Falck, Rej and Thesmar at Capital Fund Management, in 2021, took 72 published stock anomalies on CRSP and Compustat data from 1963 to 2014 and measured the decay directly. The median in sample Sharpe was 0.85. Capital Fund Management found that across 72 published strategies the median post publication Sharpe was 55 percent of the in sample figure. Publication date alone explains 30 percent of the variance in that decay and proxies for overfitting another 15 percent. Each year the decay of newly published factors rises by about 5 percentage points, which is what you would expect if the searches are getting larger and the counts still are not reported.
This is not an argument that nothing works
It would be easy to read those figures and conclude the whole enterprise is noise. The counterweight matters. Jensen, Kelly and Pedersen, in the Journal of Finance in 2023, tested 153 factors with a Bayesian hierarchical model and found that the majority replicate, cluster into 13 themes, and hold out of sample across 93 countries.
So signals exist. The point is narrower: a number found by searching is inflated by the size of the search, and the reader cannot correct for a search they were not told about. The count is half the result.
Haircut, not probability
There are two honest ways to put the count back into the number, and they are not the same thing.
Bailey and Lopez de Prado, in the Journal of Portfolio Management in 2014, defined the Deflated Sharpe Ratio. It takes the number of trials, the variance of their outcomes, the track length, the skew and the kurtosis, and returns the probability that the observed Sharpe beats the best you would expect from noise alone. A Deflated Sharpe Ratio of 0.63 means a 63 percent chance, and a 63 percent chance is not a Sharpe.
The other way is a haircut: take the observed out of sample Sharpe and subtract a penalty that grows with the log of the count, so the number you read is still a Sharpe, just a smaller one. In our canonical run the best of 48 specifications showed 0.81 walk forward and 0.63 after the haircut for 48 trials. We call that a haircut Sharpe and never a deflated one, because a desk that confuses the two will misprice the result.
Either is defensible. Printing 1.42 alone isn’t.
“Ten random configurations. Expected best in sample Sharpe 1.57. Expected out of sample Sharpe, every one of them, zero.”
What travelling means in practice
A count that lives in a researcher’s notebook doesn’t travel. It gets rounded down in the retelling, forgotten by the second meeting, and left off the pitch. For the count to matter it has to be attached to the result by the system that produced it, so nobody can detach it.
That is what the trial log is. Every specification the agent tries is a row: kept or killed, at which gate, for what reason. The survivors carry N. The Sharpe on each is printed beside a second Sharpe reduced for N. On a Research pod the ledger is shared, so five researchers on the same hypothesis count against one denominator, not five small ones. Nobody’s forty trials disappear into a folder.
You can see the mechanism on our simulator page. Set N to 8 and run. Set N to 96 and run again. The in sample number tends to rise with N, because that is what the best of a larger draw does. The haircut number doesn’t follow it. That gap is the thing the room was never shown.
| N | in sample | walk forward | haircut |
|---|---|---|---|
| 48 | 1.42 | 0.81 | 0.63 |
Illustrative output. Backtested, modelled and simulated results are hypothetical. They do not represent actual trading, they carry the benefit of hindsight, and they may differ materially from any result achieved in the market. Past performance, real or simulated, does not indicate future results.
No output of the Services is a recommendation, an offer, a solicitation, or advice about any security, strategy or account. Every investment decision is made by the customer and the customer's own investment professionals.