Quant development: Using genetic programming to create trading strategies.
We have over the last 4 years developed an automated system to create trading strategies using GP technology. The software is integrated in CUDA, so we can process data quickly using the latest graphics cards. However, we have not managed to get good results in live trading. Curve fitting is a huge problem that we have worked on for the last 2 years. A couple of weeks ago I decided to fire all my programmers, because our ideas ran out. Is there anyone out there that have any thoughts on GP and curve fitting? In theory I am convinced that GP technology is the way to go, but I simply can’t find the right setup. Do you have any ideas?
tradingsystemlab has had success with genetic programming. they have two of the top systems on futurestruth. Mike Barna is affiliated with it and has a good reputation. There may be a tip or two on their site to help. on a side note, they charge something absurd like 60k a year for their product.
indicators are fairly useless, so maybe try to add in some time of day patterns, highs and lows, and some of the non price based streams – tick index, breadth, vix, put call ratios, etc.
I agree with Murray Ruggiero that you should provide some guidance to the strategy. Personally, what i would like to do is point to a few different points on a chart that look similar, then have the algorithm go through a list of conditions (patterns, indicator thresholds, etc) to say what parameters were held in common on those bars, and create a strategy automatically. here you’d be providing guidance. if you”d like to talk about this, then definitely reach out.
you had me nodding in agreement with the comments on genetic programming until you said “indicators are fairly useless.” I think that misses the point about technical indicators. Sure, if you use them as trading rules, you’ll be disappointed, but technical indicators are really just transformations of market data. Most are designed to provide a smoothed or normalized version of the market. It’s what you do with them that matters. Saying that indicators are useless is like saying a statistical average is a worthless function. Like any mathematical transformation, indicators can be useful or useless, depending on what you do with them.
I have been writing auto trading systems for about 8 years. During that time, I have written around 40 systems that are profitable from time to time. Most of them don’t work all the time, but they all work some of the time.
The major determining factor for swing trading systems is the average movement of the market – up, down, or sideways.
For short-term (day-trading) systems, volatility is one of the keys.
As for number of parameters, my systems usually have only 2-3 input parameters. I even have a system with no input parameters at all (it’s hard-coded).
I use TradeStation and Easy Language.
I have some experience in using GP in trading. I developed a software by myself to do it and I did it in 1 month. I’m testing some strategies now in paper trading till the end of the year. before going live.
I traded in the past futures in a discretionary fashion. Where are you located at?
CUDA or not CUDA I think a good GP program can reach a good TS in about 1000 or less individuals tested at least for hourly data. Remember that the more you test the more your results are biased. (there is still some advantage anyway)
The part of GP it is only 20% of the work! after you find out the systems you have to assess it with proper tests. then you have some idea on how they can perform in the future.
I can say something about my experience and my opinion.
To have a benchmark try to use miniSP data hourly data 1999-2009. try to run it and then tell me what you found (al least sharpe ratio,profit/max drawdown)just to see what the program can find with respect to another one(the one i developed). I tested also other programs available on the net. more or less the results are there. if you want we can go private to discuss it. sharpe ratio should be >4. profit/max drawdown>6. I’m quite curious about the details and what genetic operations you used and what entry exit modes you used.
I can say that going on shorter time frames it’s very difficult the problem execution is though. 64 indicators is a very high number I have about 10 of them.
one though problem for me is data snooping! I know too well the data now. and once you have used out of sample data once, all what comeslater on after is data snooping.
about considering a posteriori the results and interpreting the results it’s a dangerous path. consider the case in which you find a TS performing good by pure lack and this has a good interpretation…you have a lucky TS with a meaning. bu it’s always a lucky TS. I would prefer to interpret the result on a very short period or another market to find an idea and then test it as a simple one.
Let’s change your problem only slightly. Let’s say that all the data series you used in your tests were randomly generated. And that you applied all or any combination of any indicators, functions or optimized parameters on these price series. All your programs would work, they would find in the data presented to them: patterns, trends and oscillators of all kinds. And would provide you with the best “trading solution” to consider going forward. I presume, from the above, that you have reached the same conclusion I did when reading on your research efforts.
The very premise of what you attempted is on shaky foundations; it implicitly states that what ever the price gyrations of the past, they will be the same in the future. So it comes as no surprise that you missed the mark. I even expect that your trading methods had a hard time beating the Buy & Hold strategy.
Go back to the basic. Start by asking questions related to the future price variations and what you would expect to extract from them. You can use past price data as a guideline to what the future may bring; but you probably won’t be able to use “the perfect set of parameters detected on the past data”. Every price series has its own signature and going forward will evolve and generate a new unique signature.
In my own research, I first designed the mathematical foundation on which to build my trading systems; this took a few years. It is only after having proven to myself that the trading methods I intended to use had a strong mathematical framework that I started simulating the implementation phase of these trading methods. It is still a work in progress.
there are better tool to “create” new trading rules as grammatical evolution. I have seen a lot of trading system but nothing works better than GE to find new rules (of course they have some modification from pure GE system).
“You digged your own grave when you mentionned this very bad assumption of yours …”
Quote=”In theory I am convinced that GP technology is the way to go, but I simply can’t find the right setup. Do you have any ideas?”
GP technology is a trendy thing that won’t still solve the problem of massive logic requested of algo trading. So in a first place you are totally wrong in choice of technology, as you already stated you are running a very intensive logic request method throughout algorythm, which require intensive CPU usage not GPU.
I am not anyone that should advise around unknown people, but it is quite funny, to notice that , not knowing anything about technology or at least the pro & con, you jumped into a conclusion and didn’t get ride of it.
Just so you know, with a correct network of CPU , you will outmatch any dumb gpu grid.
That’s why FPGA are on their own league when it concern real time algorythm.
Question: Why did you select GP as an approach? I mean it has utility is some very specific domains but it is an open question if the markets are one of them.
As an alternative that does work very well I’d suggest a predictive model using neural networks or SVM.
This is my some what same answer to one of the discussion
Though I am new on this topic so please tell me if I am wrong
First thing according to me is if you move forward using Technical analysis (TA) for analyzing markets/stocks then one should be clear about the vary definition of it . TA is a study of stock trends using historical data (using graphs) which in turn represents the mass (Human psychology) , so TA is 60% art and 40 % science . Thus the outcome of any analysis based on the study of that 40 % (science) cannot make the outcome of 85 % which is treated as one of the bast trading strategics in stocks trading
The second is that there are many trading tools (trading Indicators) available in softwares but nun of then gives complete picture stock movement alone thus one have to apply many of them to get a probable movement of stock movement
Third and the last one according to me if you analysis simple chart with simple tools it give very clear probable movement of stocks , so why to put lot of effort and money in something which is not 100 %
fourth it is important to develop a trading strategics without it you are solder in a war without weapons . but in stocks trading the most important thing according to me is the STOPLOSS , if entry in stocks and moving out of your trade or both of them you need to find where shell be my STOPLOSS (over here stploss meas stoploss (minimum risk to be taken to enter a trade )and even stoploss for securing your profit once you enter the trade)
The problem everyone is experiencing is you are trying to predict the future with historical data. The only way to predict the future is with future data. This is how the ‘real’ Wall Street makes money. Nobody bets millions of dollars if they aren’t sure they are going to win. Sad but true, guys like us can only use the standard tools of the trade, get to know individual stocks and markets intimately, and then follow the big boys when they make a more. I have a friend who went to school with a guy who now works on Wall street. They met up for a class reunion. He told him to invest in BAC. I looked at the chart. It was clearly in a down trend that no rational trader would ever go long on. I told him he was crazy. He went long and netted a great profit and got out exactly when this guy told him to (a month earlier he had this info) and the stock resumed it’s down trend. I am still developing trade algos myself, based on the tried and true standard indicators, and the trick, I have found is to understand the speed and volatility of the market to fine tune the algos based on those variables applied to standard indicators. I use a combination of RSI to get overbought or oversold general markets and MACD histograms to determine the the direction of the daily market to enter trades in individual equities that are following the general market.
What kind of results are you getting from your TA (RSI, MACD) system?
I researched TAs as model inputs some years ago, found them useless over the long term and went on to other modalities.
•Same here. Nowatzke told in some other groups and myself in private through chat inspired our team and there’s no turning back using any of the indicators. The fancier the indies, the more meaningless they are. However, as we bias toward not using any indy, we’re in the mood to listen to success stories of whom using it, just try to prove ourselves wrong. Because it’s wrong to be believe in anything to be always right. (We’re currently ranked the highest verified real money automatic trading system on FxStat. Just Google fxstat t0009. You’ll find us.)
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!