Quant development: Choosing and configuring linux for low latency?
From a Linkedin group discussion:
The often heard advice is to run trading systems on a “stripped down linux”. Does it matter what version you start with? Does this mean building your own with various features compiled out? Or does this mean turning off the mail dameon?
We are not yet doing advanced linux tuning (interrupt shielding, assigning processes to cores, etc). Is this a more important place to look?
Always start with the latest commercial distro because the latest releases are well tested and have bug fixes and new features. Off-the-shelf Linux system may not be tuned for your specific requirement. Hence it is beneficial to start with a commercial distro that closely matches your requirement and then fine tune the system in-house. Following steps is what I would recommend for general fine tuning the system for low-latency:
A. System tuning
1. Strip down the kernel :-
Though compiling-out modules and features within the kernel will mostly reduce the boot time, in some cases it will speedup the runtime as well e.g. by eliminating the inbuilt profiler on the production system etc. But it also helps to strip down the kernel of unnecessary code so that it reduces the amount of data generated during profiling.
2. Reduce noise in the userland performance data :-
Remove all the unwanted applications, packages, libraries and daemons from the system. This will reduce the data generated when doing overall system performance and also the performance data will not be contaminated by unwanted applications/tools/utilities/daemons.
3. Profile the overall system :-
Perform the profiling (e.g. Sysprof) of the whole system using all the use-case scenarios to identify the system hotspots. For this, extensive data on usecase scenarios is required so that the system can be paced through various stress levels. Perform system tuning at this stage. If the required performance benchmarks for overall system are not achieved then, depending on the profile data, decide whether to tune the kernelspace first or the userland.
B. Kernelspace tuning
1. Profile the kernel to find hotspots :-
Before doing the advanced linux tuning, understand where the kernel hotspots are on a pre-production system by using kernel profiler (e.g. OProfile).
2. Eliminate optimization issues by kernel code review :-
Review the kernel code to identify any performance related issues. Perform advance kernel tuning at this stage. Go back to step B.1 until the acceptable performance benchmarks are reached for the kernel and modules.
C. Userspace tuning
1. Fine tune the required frameworks and applications support system :-
It always helps to be as near to the metal as possible for overall performance gain. Hence reduce the latency issues in the userland by eliminating, if possible, the layers of bulky-libraries, VM etc within the frameworks and applications.
2. Fine tune the framework and applications :-
Perform the framework and application profiling on the pre-production system by using application profilers (e.g. Valgrind).
3. Review the framework and application code for identifying optimization issues :-
Review the userland code to identify any optimization issues. If the acceptable performance benchmarks for application(s) are not achieved then goto step C.1.
Go to step A.3 again.
I agree with most of what Has mentioned. However I strongly advise you not to fiddle with the kernel unless the machines you are running your production systems are only going to use software which is in house & compiled on exactly the same configuration as your production system. this has bit me once too often & is painful. In short don’t fiddle with the kernel.
is not reached the level of assigning interrupts & cores as he himself says. I don’t think that re-compiling the kernel & unloading modules will be anywhere on the horizon for the moment.
I aver that you go along with the suggestion on user space tuning. I have had lot of luck in that area.
“However I strongly advise you not to fiddle with the kernel unless the machines you are running your production systems are only going to use software which is in house & compiled on exactly the same configuration as your production system. this has bit me once too often & is painful. In short don’t fiddle with the kernel.”
When extracting every bit of performance out of the system nothing should be off-limit. And this is certainly true for low-latency systems used in trading. Whether to optimize at the kernel level depends on one’s comfort level in dealing with the kernel. But doing optimizations just within userland is not going to get you very far. Most off-the-shelf Linux distros (and even the upstream kernel) are not optimized for every hardware out there. The distros are built and configured for lowest common denominator hardware. Hence to get a low-latency system for trading will require tuning the kernel at some stage in the optimization process. But I agree, if not comfortable with the kernel then do not touch it otherwise you will have similar painful experiences..
I agree with you that nothing should be off limits. Just because you leave something on the table it does not mean that others will and the failure to invest the time and effort into tuning linux for your type of trading can have a significant impact on latency. Most off the shelf software as far as I am concerned is more about throughput than latency and using a standard distro will leave you lagging far behind.
as suggested, start with the latest stable distro as this will give you the best chance at support and allow you to benefit from the latest stable improvements. You may also want to look at the real time kernel option. In my experience, this kernel provides lower latency and more consistent performance. Note that tuning a Linux system (or any system for that matter) for a low latency trading environment will be highly dependent on your particular applications. Don’t expect a cookie cutter low latency recipe to handle all your tuning requirements. What worked for someone else may not work for you. You’ll need to spend some time in the lab with your specific application and see what tuning works best for you. Red Hat has some pretty good tuning guides that are a good starting point.
Also, when building a low latency system, you need to look at the hardware as well. You need to choose hardware that will be able to provide you with the performance you’re seeking and that will work with the version of Linux you’ll be using.
do you see any performance benefit in using RT kernel over normal ones ?
It is a big subject, however in my opinion a lot of the latency is in the network and in particularly the tcp layer. Most the easy gains are in the area of reducing the network driver interrupt coalescing so that as soon as a network packet arrives it is handled by the kernel.
—-do you see any performance benefit in using RT kernel over normal ones?”
Real-time does not guarantee low-latency. Most realtime systems will not fall into the category of low-latency systems. Hence low-latency system could be categorised as real-time systems, but not all real-time systems can be said to have low-latency. One of the characteristics of real-time systems is that it guarantees that a task is completed within a stipulated time. What that time is depends on system to system. Hence it is not averse to use RT kernel for low-latency systems, but relying solely on real-time nature of the kernel for latency improvement will not help.
There’s a lot of confusion here about “real-time”. Hard real-time systems are ones where (in the words of Fed Ex) “it absolutely positively has to be done on time” and soft real-time systems are ones where it’s important but not critical that things are done within a certain time.
In order to do hard real-time certain things need to be in the OS (e.g. unbounded priority inversion needs to be eliminated). The RT Linux kernels are designed for just that, with the goal of providing guaranteed (short) latencies for (short) time-critical operations. The approaches I’ve seen are patchy though, partly because the kernel wasn’t really designed for hard real-time systems and there is so much “stuff” in Linux that runs unpredictably.
Realtime kernels usually deliver slightly worse average latency, but if tuned very carefully can help reduce latency jitter. But I find you can get most of the benefit from careful tuning and core isolation with a non-RT kernel. RT kernels excel at dividing cpu time amongst competing demands in a predictable way. But what most people want for low-latency trading environments is to dedicate cpu cores entirely to trading threads — which is a simpler problem.
Regarding TCP latency:Solarflare’s OpenOnload kernel bypass stack can do TCP half-round-trip in 3us. Nice thing about this is that the critical path is at userlevel, so performance is the same whether you use RT kernel or not.
Yes, I have some experience of the Solarflare hardware and open onload and it does seem promising in reducing tcp latency. I have to say that It is quite a specialized solution and so you may have to adapt your OS and applications in order to fit, it may be worth it though.
In general you shouldn’t need to modify apps at all when using OpenOnload. It supports the full Linux syscall API, including all the hard stuff like fork, exec, select, poll, epoll, signals etc. We consider any incompatibilities to be bugs.
But it is certainly true that the amount of performance improvement you get is dependent on how your app interacts with the network.
It’s been my experience that the RT kernels deliver better average latency than non-RT. Yes non-RT kernels _can_ provide better latency in some cases but the latency is not as consistent. Whereas the RT kernel typically provides a tight time distribution for messages, the non-RT kernel suffers from outliers, messages that are significantly off the average. These are the messages that people will complain about. In addition to low latency, it’s important to have consistent latency.
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!