Any ideas on some good ways to test experimental automated trading systems – preferably, using a distribution-free technique (e.g. permutation tests) ?.

——

I wonder what are u want to test? What is the hypothesis?

—–

is your question about technical side — which platform/software to use, or about testing methodologies?

—–

As in optimization, or robustness evaluation?

—–

I am primarily interested in the statistical evaluation methodologies that one uses to test the validity or (statistical) “significance” of a trading rule. A monte carlo approach ostensibly, seems like a sensible approach – (typically, an MC approach involves generating a random TS and testing the rule on the series) however, my trading rules are not based directly off a financial time series, so it is difficult to see how monte carlo framework fits in my particular case.

The question then remains, how to test a rule in a robust way – so that one can ascertain that success (or otherwise) is not due to random events.

——

first of all I would suggest considering if the amount of data used in the test is statistically significant. That is, if it covers different market conditions and is simply long enough to generate at least 100 trades. Then I’d pay attention to the distribution of trade results — ideally they should be evenly distribute with no excessive outliers. In other words, if all your profit is generated by a couple of trades while other trades are around 0 I would be in doubt if the trading rule is really based on a repeating local market inefficiency and not on some occasionally caught random process.

As to MC method — I personally do not use it as I do not build mathematical models for trading systems, so either my systems provide more or less equal returns for any given period (and in this case it makes no sense to MC), or vice versa — they do use some significant information which could be simply lost when MCing. So, in both cases MC makes no sense for me. However there are a lot of people around who build their systems on mathematical models, and for them, I suppose, MC is crucial to test.

It’s a bit difficult to go any further because we don’t have any idea not only about your trading rules (and no one expects you to publish them for sure), but also about your approach: if it’s based only on abstract mathematical models or on some market facts. Your comment that your rules are not based directly on financial time series makes me think that you follow the latter approach. However any more detailed information from your side could help.

—–

thanks for your detailed response. Despite coming from a mathematical background, I believe that trying to create a parsimonous mathematical model that explains all (or even the majority) of market behaviour, is fundamentally and intrisically flawed (but thats a topic for a philosophical debate in itself).

I’m already doing most of what you recommend in your reply – including introducing a few “sample statistics” of my own. Like you, (on some reflection), a MC approach does not make sense for my approach either.

I suppose, what I’m trying to get at, is a methodology for determining how much better a particular rule is – compared to a trading rule comprising of ‘randomly generated signals’ – or perhaps a ‘trivial rule’ such as one based on SMA, Fibonnaci (or similar).

The (obvious?) approach I suppose, would be to propose null hypotheses stating that all the generated trades (from all rules being tested) are drawn from the same population (ergo; similar mean and distribution properties) and then test to see if those null hypotheses can be rejected.

Coming to think of it, that may well be the most appropriate approach (its sometimes good to think out aloud!). I would be interested though, if someone is using an alternative approach to testing whether a specific rule’s performance is largely due to chance, or due to somehow encapsulating some salient information regarding market behaviour.