Quant Development

Has anybody tried using the cloud computing to cut down the latency for quant development and HFT? Hadoop any good? I have a question, on trying to reduce the latency during the order sending process. And please feel free to correct me, if I am wrong on any part. Currently I have heard lot of the trading server use FPGA; embedding some of the arithmetic calculations in a chip to cut down the latency. I understand this must be fast since the calculations are made in the mechanical level. Not comparing to the FPGA, but recent studies in multiple industries are saying the 'cloud computing' is popularly used for low latency. 1. Would you consider this also a useful way for algo trading? 2. If anybody tried/ heard about this, would this be as fast as the FPGA? Thanks for your interest.   == since you're talking about the order sending latency, why not look at it holistically? Computational latency of the trading server may only be a fraction of network latency when your server is located far from the exchange. Computational power - wise, it depends on how heavy your order sending processing is. If its really a lot of code and you build a computer cluster with really low inter-node communication overhead, i can imagine it beating a very slow FPGA. From my knowledge of order processing complexity its very unlikely though. And of course there are other factors to consider, such as costs, business model, time, etc affecting your choice.   == We use clouds to reduce latency issues ( network/processing) with north american execution/clearing (can be trading anywhere), i.e pull lots of data within high bandwidth USA data centers and then feed processed data back to Australia.. But we dont use for order processing as these are done in real-time (FIX) and if not in real-time then latency does not matter. Typically we see host (receiving) processing as the issue not network or processor limitations ( i.e we typically hit the wall of the FIX receiver ( execution broker) to process our messages. If one developed an automated or algo order process.. hosted in the approperate cloud, then should also work.   == If someone (funding idea) decided to put a "cloud" CoLo'd in the ME data centers of the exchanges themselves, this could make more sense. In my experience the mass appeal for cloud processing would be in post trade research. Where the idea is to do huge numbers of iterations over similar/same data set to see "approximately" how your ideas would work if they would have been in play. In real time, not to mention HFT, the idea of using "cloud" computing would really only be helpful if you have no ability to have "your" processing power close to the ME, or if you were somehow getting much better quotes into this cloud then you are able to get to your own processing power. ==   Great comments,they really helped to tie up my thoughts. And allowed me to realize I was thinking in a slightly wrong direction. But got myself another idea to use this. lol However since the KOPSI derivatives market is set-up a bit different from the other international markets; I would need to analyze if the 'theory' and 'real-life' will work out. Thanks again. and wish you all a great weekend!!   == If you want to achieve efficiency from the cloud, there are some useful models. Look at the Open Data Center Alliance for some examples http://bit.ly/zvdFSK. Also, KOSCOM has discussed starting a FIX liaison group to work on low latency trading http://bit.ly/wQKprf, so you may want to reach out to them.   == I am a big fan of cloud computing and use Amazon EC2 for live execution. I haven't really found anything with lower latency combined with scalability. I dynamically add whole banks of servers to ramp up execution speed, then ramp back down, all programatically, and their latency is excellent considering you can opt for data centers closest to your execution partner. *If anyone knows better, please tell me.* Always looking for improvement.   == I wonder which studies you are referring to around cloud computing for reducing latency. To me this sounds completely wrong. At least for the latency you are focused in and that seems to be algorithmic trading latency. You might be referring though to grid computing, which can reduce latency if your algos can be parallelized. Grid and Cloud can sometimes are overlapped but in your world (or at least why understand your world is) forget about cloud (if this means going externally to do your calculations) and think about grid (if, of course you can paralellize your calculation). Hope it helps.   == Cloud is something sitting above on real hardware and in most of the cases just adding additional overheads. You must pay for virtualization using your performance. Cloud may add additional computation resources but this way you going more in increasing scalability rather decreasing latency.   == I've seen and heard of this being implemented in certain banks and prop trading shops using Hadoop. Though what I've heard is that it is in its nascent stages. Can anyone verify this?  

Get our FREE Open Source Historical Database by answering the 2 WORLD'S FASTEST TRADER/QUANT QUESTIONS