backtesting is BS and curve fitted. the best testing is live market and secondly forward testing. any other is BS
Why this strong language ? Why this anger ? You say you are a succesful daytrader. Good for you ! Stick to your strategy. But I don’t see any reason to state the things the way you are stating them.
Moreover, any new strategy has to be tested before committing real money and if you create a strategy and subsequently test on a database to see if that strategy has any validity, I do not see any reason to call this bullshit, quite to the contrary. Backtesting is the first, sensible step. After that, you run a forward test for a reasonable amount of time (depending on your trading frequency) and the final step is committing some real capital in live trading.
Ignoring these steps is what I call “imprudent” (not to use your language)
please watch your tongue and avoid meaningless posts. Remember, I am trying to keep this group clean.
back testing has lots of weakness compared to live or at least simulated live and if you dont have time of course forward testing is the least choice. Backtesting result tend to be curve fitted and the price filled is not realistic since you’re not testing it in live market so cause slippages which cause variation in P/L.
Let’s consider a “system” which buys every day at 10 AM. You account for commissions and slippage in system settings, setting them at least 1.5 times more than in reality (suppose you know the liquidity and other basic facts about the market you trade). Please tell us where the curve fitting is in this case.
I would strongly disagree. Just like with any endeavor, there are various degrees of proficiency among the participants. Curve fitting is primarily an issue if the back test/optimization is not done properly.
If you thoroughly understand how to set the in sample period vs. the out of sample periods, choose the proper optimization type based on the system, construct a valid fitness function based on the type of optimization, review the first round test results, choose between anchored and rolling walk forward analysis, perform the walk forward, analyze the results and determine the expected re-optimization window, you will have solid results that will closely match actual live trading.
Many do not do all these steps because they don’t know how and of those who do know how, many would rather live with the fantasy their system works than know that it doesn’t.
With each step mentioned above that you do not thoroughly (and I do mean thoroughly) understand, the chances of producing a curve fit system grow exponentially.
There is a huge difference in a given individual not being able to perform a function and no one being able to perform a function. For the record, I can’t dunk a basketball but that doesn’t mean no one can.
I would agree that backtesting implies curve-fitting if you just use calibration methods with black boxes and a significant number of free parameters; nevertheless, this should not be even defined as a good way of performing backtesting, in my opinion.
We can confine our attention to Fourier analysis or to any functional decomposition on a finite number of components (call them ‘basis function’, ‘neurons’, whatever) to experience over-fitting, if we only use way too many degrees of freedom without a pinch of salt.
That said, I am confident you trust Fourier analysis and-the-like in your everyday’s life. Markets are different, since we can think of data being generated by non-stationary processes; yet, in my opinion, backtesting is the only not painful means that we have to experiment with our ideas. If it is performed critically, it can provide us with a great amount of information and, above all, make evident the limits of our thought process and our assumptions.
Now, let us assume backtesting is completely unreliable (50% of times, otherwise it would give some sort of edge); what would you propose, then?
First of all , a great backtest does not mean the system will work into the future, but a bad backtest does mean you should throw away the system. In the years I have been doing this, I have seen people make comments like this and trade using charts/indicators and if you test what they do it breaks even at best. See, backtesting can tell you your baby ugly and people don’t want to hear that.
Backtesting can be abused, Back testing should be used to test a premise, not just keep trying until you find something. You want to see your premise work as expected in a backtest, not just make money. If you have a trend following premise and you win 80% of your trades over 10 years that would worry me.
The key to building system is to use the scientific method to develop and test them.
Here is the misunderstood part, slippage has nothing to do with curve fitting its just overestimation of entry and exit point, but of course slippage has to be factored in.
I am skeptical and resent those seller who always say they do backtesting with their system and then I told them the same thing that I do not believe backtesting and so I told them to do forward testing even live simulation and see the result. Most automated systems in internet are advertised bactesting and hypothetical so it means the system not performed in real situation and most ppl doesnt know what the backtesting disadvantages which is misleading the product. I am trying to protect those innocent buyers of automated system to do more research before they buy. Make sure those sellers are traders not just salesmen or researcher who doesn’t trade real live market. Further ask for their trading statements for verification and most doesnt even have it. Of course there are few out there that are legitimate and I can recommend you guys so if u want let me know.
1 day ago
That is why you must always have a “training” data set AND “validation” set to test your newly trained model. I believe stock analysis and models can be similar to process control testing, and even though we can’t predict the future disturbances in a given process, we can make our controllers (stock strategies) more robust by evaluating historical data.
im still confused as why nobody talk about forward testing. I understand nobody almost have no time for simulated real time result but at least forward testing. why so stuck up just w backtesting ? why not go forward ??
There are an uncountable number of trading systems that do not work, and some much smaller number that do. The walk forward process will help estimate the future profit and risk before trading.
Assume that a large number of alternative systems, each with its own combination of logic and parameter values, have been tested (backtested, if you will) over some historical data, say years 2000 through 2006 for purpose of discussion, calling this data period the in-sample data. Select the best of these using an objective function of your own choice, test it over the year 2007, calling this data period out-of-sample, and record the results. Reset the date range one year forward to 2001 through 2007, test all combinations, select the best, test it over 2008 and record the results. Continue stepping forward in time. If the concatenated results of the out-of-sample periods are satisfactory, consider trading the system.
Through intelligent design of the system, the backtesting can, and should, incorporate any and all features of the system, including filters to recognize market conditions where the system does not perform well, adaptation to changing conditions, inter-market relationships, commission, slippage, tests monitoring ongoing system health, position sizing, and whatever else is desired.
The trade performance over the out-of-sample period is the best estimate of future performance. Beginning to trade a system without following this procedure, or some similar, is done at the financial risk of the trader. Tomorrow is always out-of-sample. If a system has not passed rigorous out-of-sample testing in the development process, the market will perform that test for you using real money.
Given the small percentage of all possible systems that I can imagine that actually work, I will continue to backtest and validate my systems before trading them.
Thanks for listening,
In my experience all forms of testing tell you something about your strategy. While each approach has some weakness, each has value relative to how efficient it is to set up the test. I run backtests. I also run real-time (IB ‘paper’) testing. Neither will (nor can any test) tell exactly how the strategy will behave live. But you can establish a level of confidence that warrants risking real money.
Confidence is the key. Once in awhile even a simple backtest to validate logic suffices temporarily in certain maket conditions. Sometimes (typically) extensive testing just does not show enough promise; more testing won’t make the strategy better. In other words, whatever the test approach, it is just a method for incementally increasing your confidence that your strategy will not work; rarely does it work the other way.
I agree with you I think also evaluating how robust the optimization space is greatly helps evaluate how well the market will work into the future. I think Walk forward testing is a good option because it give you more out of sample trades, just having a single validation set does not work for longer term trading systems.
I agree that using process control techniques is attractive. The difficulty is that industrial processes usually have a standard deviation much smaller than the mean, while trading systems have a standard deviation much larger than the mean. (In fact, given a trading system where the standard deviation of percent profit per trade is equal to or smaller than the mean, any reasonably good trader can become extremely wealthy very quickly.) In the studies I have done, process control techniques are not sensitive enough when used with short lookback periods, and have too long a lag between point of failure and point of detection of failure with long lookback periods.