Has anyone experienced logic errors/deadlocks (system crashes) during LIVE algo trading?
Would you suggest more aggressive hedging, or running two CPUs with shared position size, notional, data? Comments are welcome
Yes I have! My recommendation is to use Amazon EC2 elastic cloud. It costs about $70 per month, and has a rock solid Internet connection. You can find instructions online.
Can you tell a bit more about your experience with cloud? I am interested to hear more.
I was doing remote trading with a prop firm, and I got really frustrated with internet reliability and system crashes. I heard about EC2 from Amazon, and I gave it a try. Basically you sign upon Amazon, then set up an ‘instance’. Once you set it up, you remotely login using window remote access. Then you load your apps remotely just like you do on your work station
Hmm, I am not hot on clouds and neither is my bank. We’re discussing suitable alternatives in the “High Frequency Trading” group.
you have to produce log or stack trace (in java use “jstack”) for your running system, then you can analyze it.
Are the crashes and deadlocks due to hardware or software problems, and is your software developed internally or externally?
If they are logic errors, as your original post suggests, then running two CPUs or in the cloud seems very unlikely to solve your problem. You’ll need to either add debugging information to track down the errors (if it’s proprietary, internal software, and you have coders at hand to do that) or bug the software vendor for a fix/workaround otherwise.
• If the problem is bugs in your code – and it must be, if you are experiencing deadlocks – then running it remotely in the cloud won’t help! You need to find the bugs and fix them. Use static analysis to find the bugs in your code *before* they cause a crash or deadlock.
I’m working on a new static analysis tool for Java that is specifically tailored to finding concurrency bugs, and am looking for potential early adopters in algorithmic trading. Please reply privately if interested
Technology issue, deadlocks typically come from hand written state engines that loop back onto existing logic. Very technology driven and even my last comment is flawed based on the number of implentation options. Is it a state engine or rules engine, given any of these at a simplistic level ignoring code errors any decision support can get stuck retesting the same premise so looking like a deadlock when infact its just happily munching away at your cpu clock cycles…. real implementation of premise based (inference engine) http://en.wikipedia.org/wiki/Drools.
Ignoring coding failure percieved dealocks can be just an inability to transition (finite state machine) or complete an inference search based on a set of rules/results that grow (inference engine)
He describes one way of getting an infinite loop. Infinite loops can look like deadlock, but real deadlock arises from mistakes in multi-threaded code.
Multi-threading is (or soon will be) unavoidable if you want speed – chipmakers are now increasing degree of parallelism (“multi-core revolution”) rather than increasing speed of processors. See http://www.gotw.ca/publications/concurrency-ddj.htm for a famous article from 2005 that explains the situation; reality since then has followed these predictions.
The problem is that it’s really hard to write correct multi-threaded code. And worse, it’s really hard to test or debug – non-deterministic behaviour (results depending on the scheduling of threads) means that getting the right answer once says nothing about it being right every time. So bugs are often very difficult to reproduce. That’s why you need static analysis, which finds bugs by analysing the code rather than by running it.
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!