Seeking quant development solutions for replay & back-testing on large sets of captured full depth market data
Seeking solutions for replay & back-testing on large sets of captured full depth market dataMy firm is looking to upgrade our proprietary trading strats/systems from using top-of-book only data to instead using full depth of book market data in the hopes it will help us make more informed trading and order sizing/routing decisions and are interested in hearing other people’s experiences making such a transition, and in particular what solutions are out there for replaying full depth data in our research and backtesting environment. We have been capturing the depth data from all “lit” N.American Equity Exchanges for over a year and now have over 5TB of it captured in binary format. Ideally we would like to be able to utilized our existing captured data and replay it on demand in an efficient way without having to put much time/resources towards developing our own system to do so. What sorts of solutions are others using to do this? Interested in both hardware and software aspects, with a priority towards the depth data replay/backtesting capabilites, however if the system(s) can also be used in low latency realtime production environment, and/or have reporting or CEP capabilities that can aid in R&D, that certainly doesn’t hurt.
==-Have you looked at OneTick? My company produces SSD based hardware to act as either the application engine for these types of applications, or we can also configure as iSCSI storage for another server and provide over 3GB/s bandwidth and up to 600,000 IOPS from a single 2u box at under 500Watts. 10TB of SSD storage per system. Let me know if you would like more data or check out our url @ —
For speed and CEP, You might have a look on these ones : -StreamBase -TimesTen -kdb+ -Sybase ASE -Gemstone, Vhayu (mentioned too but We never tried these two last) You might need a bit of TIME to assess ,choose and setup the whole thing
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!