Tag Archives: genetic programming

Need decent genetic programming Python packages

Need decent genetic programming Python packages

https://github.com/trevorstephens/gplearn

https://www.quora.com/What-are-some-good-genetic-programming-libraries-in-Python

Not practical for TensorFlow

Genetic algorithm exemple for Tensorflow? from MachineLearning

 

Note this paragraph:

  • Machine learning/artificial intelligence – Machine learning algorithms have become more prevalent in recent years in financial markets. Classifiers (such as Naive-Bayes, et al.) non-linear function matchers (neural networks) and optimisation routines (genetic algorithms) have all been used to predict asset paths or optimise trading strategies. If you have a background in this area you may have some insight into how particular algorithms might be applied to certain markets.

https://www.quantstart.com/articles/How-to-Identify-Algorithmic-Trading-Strategies

https://www.quantstart.com/articles/parallelising-python-with-threading-and-multiprocessing

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Speedy Python with Numba and Genetic Programming

Speedy Python with Numba and Genetic Programming

A comment came in from highly active newsletter subscriber:

Python slow and inefficient”

 

You mention Cython in speeding up your code, you may like to consider for certain situations adding numba to your arsenal as well. Numba allows you to keep your python code in generic form and use a decorator to invoke a LLVM (JIT)compiler to compile the byte code run close to machine code times. http://numba.pydata.org/numba-doc/dev/user/examples.html

 

I use this often after profiling the python code to target bottlenecks, (http://markdewing.github.io/blog/posts/prototype-for-profiling-python/)

It can be used with multithread and also Numba Pro and CUDA with Nvidia GPUs. By profiling the code you can ascertain where the bottlenecks are in the code and where to target your efforts in speeding up code.

https://gmarkall.github.io/tutorials/pycon-uk-2015/#12

 

For comparison studies see https://www.ibm.com/developerworks/community/blogs/jfp/entry/How_To_Compute_Mandelbrodt_Set_Quickly?lang=en

 

(Another option for fast computation is to write a Fortran function directly ( yes I am of that vintage! J ), and use the f2py package to interface with the function.)

 

The other comment: you refer to the issue of the “market landscape always changing”, and you need a library of algorithms to cater for this.  I would add not only a library, but a “dynamic” library. One such approach would be to use Machine learning to generate new ideas constantly. Genetic Programs (GP) can generate trading ideas or systems from the data itself (more advanced applications will write the system code) , you may have a machine dedicated solely for this task. If you map your search space to a higher dimension you may discover patterns  that are exploitable and give you an edge. Being nimble counts.

 

I followed up with questions of:

What GP Python packages to use? Compatible with Machine Learning Python packages that this can work with? Any such examples for this use?

This story shall continue

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Help with how Trading Systems Labs says machine learning and genetic programming does work for quant and market forecasting

I posted this link a few days ago:

robot

Trading Systems Labs says machine learning and genetic programming does work for quant and market forecasting – See more at: https://quantlabs.net/blog/2014/06/trading-systems-labs-says-machine-learning-and-genetic-programming-does-work-for-quant-and-market-forecasting/#sthash.my7SoGhu.dpuf

 

Here is some commentary from someone at this link:

 

I used Machine Learning / GP to develop the strategies that I use every day to trade my personal accounts and my clients money. ML/GP can work, and in fact works very well for me. – See more at: https://quantlabs.net/blog/2014/06/trading-systems-labs-says-machine-learning-and-genetic-programming-does-work-for-quant-and-market-forecasting/#sthash.my7SoGhu.dpuf

 

 

I don’t think anyone with truly good working strategies is going to show you or give you the code. Makes no sense. ML/GP is just a tool anyhow. No holy grail. My concern is not what has worked in the past, but what is currently working, and most importantly, having the skills and tools to stay profitable going forward. – See more at: https://quantlabs.net/blog/2014/06/trading-systems-labs-says-machine-learning-and-genetic-programming-does-work-for-quant-and-market-forecasting/#sthash.my7SoGhu.dpuf

 

I am looking for anyone who has tutorials that covers this area. I know of Matlab Machine Learning Toolboxand other techniques but if you have a really decent tutorial that works, let me know as I want to talk.

 

If proven, I will gladly post my findings for my Members at:

 

The affordable one http://quantlabs.net/membership.htm

 

And the Elite one http://quantlabs.net/academy/be-an-elite-quant/

 

Thanks Bryan

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Profit with genetic programming with news analysis and context contingency with fuzzy weighting

Profit with genetic programming with news analysis and context  contingency with  fuzzy weighting

 

Search my blog for this headline:

>>

>> When you check out these sort of links below, you could save

>> yourself many months to years of wasted effort on potential trading

>> ideas that go nowhere.

>>

>> [1]

>>

>> _Genetics programming does not work from my POV regardless of what

>> academics it but what about profitable real world trading

I got a response from :

> Agreed.  One however though, as it is slow to correct for

> parametrization – in cases where there is a lot of random volatility

> noise – it can aid in long term value paths by sorting for selection

> of choice over “stay the course” potentials, probably not too useful

> in quant but in adjunct areas like news analysis and context

> contingency potentially valuable in long run models if hybridized with

> fuzzy weighting for other methods …  possibly.

I just like using genetic algorithms. Just another way of what you just said. ->

Bryan

Speak out by joining my FREE newsletter 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

HFT is here! 3 videos posted demoing FIX, Matlab, genetic programming, moving average, RSI, etc

Hi there
This has been a long time coming but what you see below is why the QuantLabs.net Premium Membership fees are going up for new members!
It is here. Finally! The HFT demos I have been talking about have been posted:
1. Youtube video demo of HFT system with Matlab, Java, IQFeed data capture to analytics to order execution using FIX
2. Youtube Demo of HFT Matlab system with evolutionary learning aka genetic algorithm signal and indicator of RSI Moving Average
3. Youtube Demo of HFT system in Matlab with moving average strategy with plots
Got questions on this Premium membership? Ask live in our upcoming live webinar on Mon Jan 7 6 PM EST.
Thanks Bryan
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Quant development: Using genetic programming to create trading strategies

Quant development: Using genetic programming to create trading strategies.

We have over the last 4 years developed an automated system to create trading strategies using GP technology. The software is integrated in CUDA, so we can process data quickly using the latest graphics cards. However, we have not managed to get good results in live trading. Curve fitting is a huge problem that we have worked on for the last 2 years. A couple of weeks ago I decided to fire all my programmers, because our ideas ran out. Is there anyone out there that have any thoughts on GP and curve fitting? In theory I am convinced that GP technology is the way to go, but I simply can’t find the right setup. Do you have any ideas?

 

tradingsystemlab has had success with genetic programming. they have two of the top systems on futurestruth. Mike Barna is affiliated with it and has a good reputation. There may be a tip or two on their site to help. on a side note, they charge something absurd like 60k a year for their product.

indicators are fairly useless, so maybe try to add in some time of day patterns, highs and lows, and some of the non price based streams – tick index, breadth, vix, put call ratios, etc.

I agree with Murray Ruggiero that you should provide some guidance to the strategy. Personally, what i would like to do is point to a few different points on a chart that look similar, then have the algorithm go through a list of conditions (patterns, indicator thresholds, etc) to say what parameters were held in common on those bars, and create a strategy automatically. here you’d be providing guidance. if you”d like to talk about this, then definitely reach out.

 

you had me nodding in agreement with the comments on genetic programming until you said “indicators are fairly useless.” I think that misses the point about technical indicators. Sure, if you use them as trading rules, you’ll be disappointed, but technical indicators are really just transformations of market data. Most are designed to provide a smoothed or normalized version of the market. It’s what you do with them that matters. Saying that indicators are useless is like saying a statistical average is a worthless function. Like any mathematical transformation, indicators can be useful or useless, depending on what you do with them.

 

I have been writing auto trading systems for about 8 years. During that time, I have written around 40 systems that are profitable from time to time. Most of them don’t work all the time, but they all work some of the time.

The major determining factor for swing trading systems is the average movement of the market – up, down, or sideways.

For short-term (day-trading) systems, volatility is one of the keys.

As for number of parameters, my systems usually have only 2-3 input parameters. I even have a system with no input parameters at all (it’s hard-coded).

I use TradeStation and Easy Language.

—–

I have some experience in using GP in trading. I developed a software by myself to do it and I did it in 1 month. I’m testing some strategies now in paper trading till the end of the year. before going live.
I traded in the past futures in a discretionary fashion. Where are you located at?
CUDA or not CUDA I think a good GP program can reach a good TS in about 1000 or less individuals tested at least for hourly data. Remember that the more you test the more your results are biased. (there is still some advantage anyway)
The part of GP it is only 20% of the work! after you find out the systems you have to assess it with proper tests. then you have some idea on how they can perform in the future.

I can say something about my experience and my opinion.
To have a benchmark try to use miniSP data hourly data 1999-2009. try to run it and then tell me what you found (al least sharpe ratio,profit/max drawdown)just to see what the program can find with respect to another one(the one i developed). I tested also other programs available on the net. more or less the results are there. if you want we can go private to discuss it. sharpe ratio should be >4. profit/max drawdown>6. I’m quite curious about the details and what genetic operations you used and what entry exit modes you used.

I can say that going on shorter time frames it’s very difficult the problem execution is though. 64 indicators is a very high number I have about 10 of them.

one though problem for me is data snooping! I know too well the data now. and once you have used out of sample data once, all what comeslater on after is data snooping.

about considering a posteriori the results and interpreting the results it’s a dangerous path. consider the case in which you find a TS performing good by pure lack and this has a good interpretation…you have a lucky TS with a meaning. bu it’s always a lucky TS. I would prefer to interpret the result on a very short period or another market to find an idea and then test it as a simple one.

 

Let’s change your problem only slightly. Let’s say that all the data series you used in your tests were randomly generated. And that you applied all or any combination of any indicators, functions or optimized parameters on these price series. All your programs would work, they would find in the data presented to them: patterns, trends and oscillators of all kinds. And would provide you with the best “trading solution” to consider going forward. I presume, from the above, that you have reached the same conclusion I did when reading on your research efforts.

The very premise of what you attempted is on shaky foundations; it implicitly states that what ever the price gyrations of the past, they will be the same in the future. So it comes as no surprise that you missed the mark. I even expect that your trading methods had a hard time beating the Buy & Hold strategy.

Go back to the basic. Start by asking questions related to the future price variations and what you would expect to extract from them. You can use past price data as a guideline to what the future may bring; but you probably won’t be able to use “the perfect set of parameters detected on the past data”. Every price series has its own signature and going forward will evolve and generate a new unique signature.

In my own research, I first designed the mathematical foundation on which to build my trading systems; this took a few years. It is only after having proven to myself that the trading methods I intended to use had a strong mathematical framework that I started simulating the implementation phase of these trading methods. It is still a work in progress.

 

there are better tool to “create” new trading rules as grammatical evolution. I have seen a lot of trading system but nothing works better than GE to find new rules (of course they have some modification from pure GE system).

 

“You digged your own grave when you mentionned this very bad assumption of yours …”

Quote=”In theory I am convinced that GP technology is the way to go, but I simply can’t find the right setup. Do you have any ideas?”

GP technology is a trendy thing that won’t still solve the problem of massive logic requested of algo trading. So in a first place you are totally wrong in choice of technology, as you already stated you are running a very intensive logic request method throughout algorythm, which require intensive CPU usage not GPU.

I am not anyone that should advise around unknown people, but it is quite funny, to notice that , not knowing anything about technology or at least the pro & con, you jumped into a conclusion and didn’t get ride of it.

Just so you know, with a correct network of CPU , you will outmatch any dumb gpu grid.

That’s why FPGA are on their own league when it concern real time algorythm.

—–

Question: Why did you select GP as an approach? I mean it has utility is some very specific domains but it is an open question if the markets are one of them.

As an alternative that does work very well I’d suggest a predictive model using neural networks or SVM.

 

This is my some what same answer to one of the discussion

Though I am new on this topic so please tell me if I am wrong

First thing according to me is if you move forward using Technical analysis (TA) for analyzing markets/stocks then one should be clear about the vary definition of it . TA is a study of stock trends using historical data (using graphs) which in turn represents the mass (Human psychology) , so TA is 60% art and 40 % science . Thus the outcome of any analysis based on the study of that 40 % (science) cannot make the outcome of 85 % which is treated as one of the bast trading strategics in stocks trading
The second is that there are many trading tools (trading Indicators) available in softwares but nun of then gives complete picture stock movement alone thus one have to apply many of them to get a probable movement of stock movement

Third and the last one according to me if you analysis simple chart with simple tools it give very clear probable movement of stocks , so why to put lot of effort and money in something which is not 100 %

fourth it is important to develop a trading strategics without it you are solder in a war without weapons . but in stocks trading the most important thing according to me is the STOPLOSS , if entry in stocks and moving out of your trade or both of them you need to find where shell be my STOPLOSS (over here stploss meas stoploss (minimum risk to be taken to enter a trade )and even stoploss for securing your profit once you enter the trade)

—-

The problem everyone is experiencing is you are trying to predict the future with historical data. The only way to predict the future is with future data. This is how the ‘real’ Wall Street makes money. Nobody bets millions of dollars if they aren’t sure they are going to win. Sad but true, guys like us can only use the standard tools of the trade, get to know individual stocks and markets intimately, and then follow the big boys when they make a more. I have a friend who went to school with a guy who now works on Wall street. They met up for a class reunion. He told him to invest in BAC. I looked at the chart. It was clearly in a down trend that no rational trader would ever go long on. I told him he was crazy. He went long and netted a great profit and got out exactly when this guy told him to (a month earlier he had this info) and the stock resumed it’s down trend. I am still developing trade algos myself, based on the tried and true standard indicators, and the trick, I have found is to understand the speed and volatility of the market to fine tune the algos based on those variables applied to standard indicators. I use a combination of RSI to get overbought or oversold general markets and MACD histograms to determine the the direction of the daily market to enter trades in individual equities that are following the general market.

—–
What kind of results are you getting from your TA (RSI, MACD) system?
I researched TAs as model inputs some years ago, found them useless over the long term and went on to other modalities.

 

•Same here. Nowatzke told in some other groups and myself in private through chat inspired our team and there’s no turning back using any of the indicators. The fancier the indies, the more meaningless they are. However, as we bias toward not using any indy, we’re in the mood to listen to success stories of whom using it, just try to prove ourselves wrong. Because it’s wrong to be believe in anything to be always right. (We’re currently ranked the highest verified real money automatic trading system on FxStat. Just Google fxstat t0009. You’ll find us.)

 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!