Algorithmic trend detection methods
Trend detection is the corner-stone of many trading strategies. In my research I have found that accurate classification of the current trend (up, down, sideways) is as important as the entry/exit rules on which strategies are based. Currently I am using a naive approach to trend detection – the gradient (or 1-period change) in a 50 period simple moving average with surprising levels of success. What other methods have you used or where would you recommend I look for more advanced methods?
Have a look at ADX + DMI
Thanks, that’s actually quite good. What I’m interested to do is test how well a trend indicator classifies a trend and then use a trend detector in various algorithms.
I suppose I could do one of two things.
1. Use ADX/DMI in a strategy and optimise for maximum PnL / drawdown by variing the the “trending” or “consolidating” threshold (25 in that article) as well as observing the crosses
2. Some other method to test how well ADX/DMI classifies a trend (up/down/sideways). Not sure how to do this at all but would be the most scientific method.
What do you think?
check out http://wl4.wealth-lab.com/cgi-bin/WealthLab.DLL/getpage?page=ChartScriptSearch.htm .
Make a code search for ADX and you should find over 100 scripts using ADX in the decision making process. There should be enough there to give you food for thought on the use of ADX.
I’ve read the explanation of ADX fromhttp://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:average_directional_index_adx and it looks way to complex to be optimal. Alarm bells for me are:
* dividing by the “True Range” which is not defined (presumable the range over two days)
* use of Simple moving averages (c.f. something better like exponential weighting)
* it’s too complex for Investopedia to give the algorithm
Personally I’d convert (high, low, open, close) into a volatility estimate and compare the likelihood of the observations (under an exponential decay window) fitting the mean slope verses the random walk you’d get from no trend.
* the market, as we can model it, is largely a random walk, the signal we want is a small drift and is very small compared to the noise if the random walk
* given massive noise, you can get days where the range is small, but that’s just because the random walk happened to go nowhere, it doesn’t indicate significant movements in itself
* we’ve no good definition of “trending” but at the very least we mean that the movement is more than we’d expect from random fluctuations alone, so it makes sense to compare to this null hypothesis
I’d be happy to work with people on this idea.
very interesting, and scientific not pseudo-scientific (which is the feeling I get from many technical indicators). That being said all methods will have parameters. ADX will take the True Range over a window and I’m guessing your method would also use a fixed window with which to do comparisons?
In any case I’d certainly be interested to hear more.
The ADX indicator tries to evaluate the “mood” of the price movement. It is not a precise instrument but can give a relatively good “feel” of what prices have been up to the now. It is not a forecasting tool, I would say rather that it is an observation tool. It gives a general sense of market pressures and tries to show visually an approximation of trade imbalances. By its calculations, (the formula is available all over the net), you will have the +DI > -DI if on average the price movement has been above the price mid-range over the period of interest. And this is interpreted has having more buyers than sellers; at least more trades going on above mid-range.
The interesting part is the above mid-range; if you are playing an up trend then the ADX could be viewed as an added market factual opinion on where the price is. And playing an up trend it should be preferable to have some indicators that could confirm that indeed, what you are looking at, has the same behavior as an up trending price.
your observations on market noise are very relevant, the signal in price movements is very weak. In fact, when one considers the market’s secular trend which has stood around 10% over the last 200 years in the US market; one should also realize that on a $50 stock, this long term trend amounts to just 2 cents per trading day. So indeed, the signal is drowned in all the noise, all the randomness. A 1% move in price on this $50 stock (50 cents) would drown the signal in 96% noise.
I just finished part 1 of my latest article on this very subject. You might be interested, it is available here: http://alphapowertrading.com/index.php/blogmenu/81-on-seeking-alpha
And I would also be interested in developing your concept. In some ways, your volatility estimates do behave about the same as the ADX.
Chuck LeBeau actually published a book years ago where he spends a good portion of it discussing the uses of ADX and DMI – it’s still one of the best write ups out there IMHO. Also, Linda Bradford Rasche published her thoughts on it in several trading magazine interviews. One observation she made was that although ADX DMI can be accurate and smooth, it is also highly lagged, so in faster markets or higher frequency timeframes, it could signal a trend change just as the trend is ending. I think I remember she used a 14 period ADX and a 28 period DMI +/- setting, which is still fairly long, but you can shorten it while keeping the relationship the same (i.e. 7 pd / 14 pd, etc.).
I think I just wrote in another post that I really like using Keltner channels in place of moving averages – they seem to reduce whipsaws (not going to repeat myself here). Also, I had some luck a few years ago using a combination of Linear Regression Slope and R-squared technical indicators; R-squared can indicate the start of a trend, whereas LRS can indicate direction.
Finally, my absolute favorite for trend detection, unoptimized and with a variable lookback period (rather than the typical fixed period indicator) is the Random Walk Index – I believe Poulos was the author and it was originally published in Technical Analysis of Stocks and Commodities magazine.
Yes, in the simplest incarnation there will be a fixed range over which the random walk test is evaluated over. However, if it’s set up right then this range itself can be a parameter over which you do a significance test (e.g are we trending over the last 20, 30 days). Or it may be that there are practical considerations (e.g. trading costs) that limit the range over which it’s sensible to look for trends.
My personal position is that I’m not interested in signals that can give a “mood” or a “feel” – I’m only interested in building completely automatic systems. Of course many noisy indicators may be combined to make one that is less noisy, but I still want no human in the loop.
Ondrea: Thanks for the pointers to the Random Walk Index, the idea behind it is what I want to do, but it’s not the implementation that I’d choose (it still uses True Range, which I have found a definition here http://www.linnsoft.com/tour/techind/trueRange.htm)
BTW I’ve just come across this list of signals which looks interesting:http://www.linnsoft.com/tour/technicalindicators.htm#TIList
You may find it surprising but none of the large trend followers would use any exit/entry rules. They are quite open about this. There are various problems aligned with them: capacity and non-robustness (as your events might be far too infrequent to make good stats).
Here’s what I would describe as a modern predictor:
A predictor is acting on volatility adjusted returns to predict the next volatility adjusted return (e.g. Sharpe ratio). Any estimate you have done can be compared with the realized Sharpe ratio (although measuring the volatility is tricky). Following this path you can build a more Bayesian framework and self-correcting indicators.
Your positions will essentially align with the strength of your predicted Sharpe ratio.
I read your posts with interest. I have to say I have been using some of the mentioned approaches myself over the years. However I currently use a totally different approach based on the COT (committment of traders) on a weekly basis and is impressive to see how trends are predicted rather than detected! Anyone else?
—I don’t understand your comment on trend followers not using entry/exit rules. Your last sentence seems to be entry/exit rules for trend following (for some implementation of “align”).
Working a bit more on the maths I was thinking of, the likelihood of getting a certain price movement from just a random walk is a Gaussian distribution. If we assume (as is likely) that the drift component is small, then the likelihood of getting a certain price movement assuming a small drift is also a Gaussian with offset mean but pretty much the same variance . The likelihood ratio is one Gaussian over the other which (assuming equal variances) simplifies to exp(drift^2/variance) or exp(sharpe^2). So predicting volatility adjusted return/Sharpe ratio is essentially the same idea.
Interestingly, the chance of loosing money monotonically decreases with increasing Sharpe ratio, which is a pretty nice property to have.
with entry/exit rules I meant a set of rather old-fashioned rules (but still used by a lot of funds) that include strategies such as
“Enter position when two moving averages cross and keep it ’til they cross again” or “Sell everything when price falls below x”..
Such events are rare and if they appear it might be far too expensive to close the position (market impact). A mathematicial problem of such rules is that your Sharpe ratio surface (I tend to think of strategies as mappings asigning to each parameter configuration a Sharpe ratio) is getting discontinuous.
Yes, predicting a volatility adjusted return and are Sharpe ratio is indeed the same. I have tried to state this in my previous comment. Once you scale your position with exp. Sharpe your exp. Profits scale quadratically with the exp Sharpe 🙂 The more Sharpe, the better…
Thanks for the clarification on entry/exit rules.
I meant that my idea of likelihood ratios came down to the same as your volatility adjusted return/Sharpe ratio. So there’s not a lot of point in pursuing my idea further.
Are you saying that with a certain number of assets to invest in you should allocate according to Sharpe ratio? I’d argue that it should be sharper than Sharpe (if you’ll forgive the expression), that is if APLC has a Sharpe of (say) 1 and BPLC has a Sharpe of 2 then I’d allocate more than 2/3 of the assets to BPLC. But I don’t have a mathematical proof of this and of course it depends on how good your prediction of Sharpe is and how correlated the assets are. The closest I can come to saying how much sharper is that you should maximise the expected Sharpe ratio of the mixture.
Right, I get it – so portfolio optimisation/rebalancing based on the computed Sharpe of the underlying asset? Whats the rebalancing period? I also suppose there must be a risk model that oversees re-balancing decisions else it would put 100% of your money in NYSE:GLD 😉
Very interesting idea. Yes I have noticed that rules based is far from optimal and to be an effective quantitative trader and solve the non-linear problem you’re going to need a non-linear solution.
COT reveals something about existing traders’ positions hence big money sentiment. Yes I’m not surprised there’s a link to that and subsequent trends. Also Call/put ratio, VIX, Bullish Percent index, Trader sentiment surveys (American investors) show a little about sentiment hence are worth investigating as leading indicators.
Thanks for your suggestions, particularly lists of indicators and other methods for detecting trend. Keep the ideas coming, very useful!
Yes, I think that expected Sharpe should be your base measure because:
* if you can borrow money/everage it gives you the greatest returns
* it minimises the expected drawdown – which is psychologically very important
* (as I’ve just proved) it measures the probability that there is an underlying increase not just a random walk
We haven’t answered your question as to what should be the window function on Sharpe – I’ll leave that for others to contribute.
Rather than think of a rebalancing period it’s better to think in terms of cost of rebalancing. If the cost is negligible then do it every day (or every tick), and if it costs a lot (say 0.5% or more) then only do it when the expected gains outweigh the expected transaction costs.
And don’t knock the portfolio that was 100% in Gold, you could have made a very decent return this year. The trick is to get out when you think the price movement is going to be against you, which is what this thread is all about.
there is no such thing as an optimal window length to estimate expected Sharpe ratios. You can run several estimators and weigh them accordingly to their recent success. However, whenever you rebalance your portfolio you have probably a new opinion about your Sharpe (otherwise you wouldn’t rebalance I hope). Also I don’t think a fund manager has an investment horizon? Investors have, but you have all sorts of them. A standard CTA would update positions once on a daily basis and would therefore update Sharpe ratios daily, too.
Tony: If you believe a bit in Markowitz and you believe that there are no correlations its indeed a good idea to do the 2:1 split as in your example. The expected profit for your position p (say number of shares or contracts) in an asset is
PROFIT = p * volatility * expected Sharpe
I think of risk position r = p * volatility.
Markowitz in its most simple form tells you therefore r ~ inv(Correlation Matrix) * expected Sharpe. If your assets are uncorrelated… eh, voila.
Of course you can pimp your utility function and include penalties for updating positions, etc. I am a big fan of sparse portfolio updates using the L1-Norm (papers by Daubechies…).
Your portfolio would only be 100% in Gold if all other assets had expected Sharpe ratio 0. Rather unlikely I would say.
However, I apologize for deviating from the original question. All I really wanted to say is that I don’t believe in those golden exit or entry rules. A position is ultimately not a step function with a few different values. A position is a very dynamic time series that reminds me of the shape of those beautiful mountains surrounding my Swiss home. It goes up and down. The expected Sharpe ratio is the main driver though.
Trend Detection — Of course this depends on your time-horizon, by changing to a non-time based bar such as constant-range-bars smooths out your indicators and gives you a more controlled transition from one regime to the next.
Portfolio Rebalancing — There was recently a study done showing that surprisingly, weekly rebalancing performed nearly as well as daily, and monthly wasn’t that far off either. So weekly isn’t a bad compromise. When you factor transaction costs there is a significant advantage.
Side Note: The above rebalancing frequency strictly depends on your algorithm and the underlying volatility, but the study was done on equities with traditional modern portfolio theory.
I also like linear regression slope and LSMA.
Thanks, but I lie closer to “everything is correlated” than to “everything is uncorrelated”. I also don’t think you can simply estimate the correlation matrix (too many parameters and correlation is dependent on volatility), but yes, under your assumptions there is a simple answer. And of course it extends to whatever model of correlations you use.
But, as you say, back to the question. I’d agree that running several Sharpe estimators and using them to estimate where the trend started is the right way forward.
I know people that ignore correlations without being aware that they have set them implicitly to zero which is a rather strong bias. There are tons of academic literature on the estimation of correlation matrices (DCC models by Engle are my choice) which are ignored in industry. I think we both know the concept of a convex combination between the two lies you have indicated 🙂
I now post my TRADING ALERTS
into my personal FACEBOOK ACCOUNT
. Don't worry as I don't post stupid cat videos or what I eat!