Top mistakes to avoid using quant analytics

(Last Updated On: April 23, 2012)

Top mistakes to avoid using quant analytics

Hi all. I’m cross posting a question that I put on a similar group to see whether folks here are also interested in throwing out some answers…

I’ve been thinking about this for a little bit, and wondered what some of you thought about this subject. I am interested in polling folks for what they think the most important “schoolboy errors” that people make when moving into this space?

I am thinking along lines of
1 – overfitting data
2 – poor backtesting procedures (no in/out sample + no costs)
3 – making inappropriate use of future information without realising it…

Things like that. I could add a few more, but I am more interested in seeing what comes up in the discussion…

I would be interested in your experiences in this area…



hmmm… looks like mistakes are as carefully guarded as the successes 😉



Some strategies in equities and currencies tend to fade in and out in success i.e. they stop being successful and mysteriously start being successful again. So back testing has to be routinely carried out, depending on the trading time frames.
The back testing should also consider breaking down the trading day, i.e. Asian, UK, US sessions, market closing and openings. These different times have different participants and hence different strategies come to play.
The Asian session is notorious for low volume and cheeky brokers stop hunting, best avoided!
These are a few of my head that I know that are really important.
I apply these to trading they hold equally for algo trading, the only difference being the algo has no emotion



Data snooping, a.k.a. oversearching (notice that this is not the same as overfitting). White’s reality check or k-fold cross validation are the guards I use against it.


I believe the three points you’ve made are spot on.

I’d add to this list a failure to attribute performance.
If a stretegy has some theoretical basis then it should be verified that the results comply with what is theoretically expected regardless of what sort of Sharpe ratio it produces over a certain data sample.
If a model consists of multiple components (for example signal, portfolio, risk etc) then the contribution of each of them must be carefully examined.

A little example that I really like is a fair game of tossing a coin.
This game might be overquoted but I still find the display fascinating.
And this is directly related to “strategies mysteriously becoming successful after periods of non-performance” as mentioned by Zaki above.
If you have not looked at it before then just execute this in R repeatedly.
plot(cumsum(sign(runif(10000) – 0.5)), type = “l”)

I bet you that within a dosen or 2 of runs you will see a very convincing “equity curve”.
And that’s with 10000 points ~ 40 years if you assume you play this game once a day.
In other words if you don’t know/care where the performance is coming from then the dangers of having a completely random positive PnL are very real whether it’s backtested or live.

Also I’d make point 3 stronger. There’s no _appropriate_ use of _any_ future information.



Since the question was about “people moving into this space”, I would argue the #1 mistake is thinking this is easy – or even doable – for most people it never will.

The #2 mistake is to move into this space with under-par knowledge & skills.

The #3 mistake is to start looking for an edge (& developing a system) without a clear understanding of what it will take to ensure statistical significance of the result.

The #4 mistake is to use the wrong tools … accurate historical data, effective trading system development & execution platform, reliable real-time datafeed easily come to mind



Thinking that your Internet connection and PC fast enough for good execution time for trades. What means better use VPS for Forex trading. Here are 4 reasons why: http://pipburner.com/4-reasons-why-you-should-use-a-vps-for-forex-trading/



IMO the biggest mistake is over-belief in math. By that I mean, looking at backtest results without understanding what those results are saying about market structure, and consequently not understanding when, in the live test, the strategy makes sense to actually deploy.

Then there are all kinds of technical details related to trying to run too fast for the tools – eg looking at 1 second bars on a retail data feed.


NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Subscribe For Latest Updates

Sign up to best of business news, informed analysis and opinions on what matters to you.
Invalid email address
We promise not to spam you. You can unsubscribe at any time.


Check NEW site on stock forex and ETF analysis and automation

Scroll to Top