Tag Archives: latency

For HFT: Lowest latency with sockets over APIs? How to build a C++ Linux based open source FIX engine

Hi there

Another opportunity, get in the on the action soon as we are closing out this FREEMIUM stuff soon! In other words, join the QuantLabs.net Premium membership while you can for the upcoming private webinars. .

I went to my Meetup last nite where I always learn lots of things. From what I see, everything is being streamed into trading platforms these days. Also, screw APIs as they becomes bottlenecks with latency so you need to learn how to program sockets.  Interactive Brokers comes to mind with their Trader Workstation  versus their FIX gateway options.
Not only that, many in the past have told me that using R with RCpp/RInside could be a bottleneck. There is a potential with that so it leads me into investigating event rule based programming using a Complext Event Processing engine like Esper. Well, there is none for C++ as Esper is for C# or Java. Too bad but there is an option to stream it in. I posted about that yesterday. There are some options I posted at:
I am investigating 2 options currently.
I also got my backups in place if R does indeed become the bottleneck, I can hint it is most popular part of QuantLabs.net. It also involves a very big expense but you gotta do what you gotta do to get things working. Aahh… the joyous challenges in working on these HFT potential platforms?
Another important element are FIX engines built on top of wonky QuickFIX. I documented my experience on one solution called FIX8
These are the most complicated parts of any HFT platform. As I am building this out, I am realizing the incredible value people are getting being on this list. 

JOIN NOW 

Benefits are listed are here. 
Bryan
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

LMAX How to get 100K transactions per second in less than 1 ms latency

LMAX How to get 100K transactions per second in less than 1 ms latency

infoq.com

Martin Thompson and Michael Barker talk about building a HPC financial system handling over 100K tps at less than 1ms latency by having a new approach to infrastructure and software. Some of the tips include: understand the…

 

==

That was like 18 months ago, wasn’t it?

 

Yes, this is an intro served for the beginning level folks in the group. Not everyone is this group is at the same level. I am trying to bring everyone up to speed

infoq.com

Martin Thompson and Michael Barker talk about building a HPC financial system handling over 100K tps at less than 1ms latency by having a new approach to infrastructure and software. Some of the tips include: understand the…

 

That was like 18 months ago, wasn’t it?

 

Yes, this is an intro served for the beginning level folks in the group. Not everyone is this group is at the same level. I am trying to bring everyone up to speed

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

LMAX Disruptor model gets less than 1 ms latency for quant development and HFT? Video using open source Java, JBOSS, PostgresSQL

LMAX Disruptor model gets less than 1 ms latency for quant development and HFT? Video using open source Java, JBOSS, PostgresSQL

LMAX – How to Do 100K TPS at Less than 1ms Latency

Summary
Martin Thompson and Michael Barker talk about building a HPC financial system handling over 100K tps at less than 1ms latency by having a new approach to infrastructure and software. Some of the tips include: understand the platform, model the domain, create a clear separation of concerns, choose data structures wisely, and run business logic on a single thread.

http://www.infoq.com/presentations/LMAXgen

Also, the one thing I like about this talk is how they pre-allocated their buffers to optimize and minimize garbage collection. They put everything in a ‘ring buffer’ to minimize GC and optimize their caching. This may eliminate those who doubt Java because of garbage collection.

This has been open sourced as well http://code.google.com/p/disruptor/

After researching, this may become a game change for my solutions?

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

lowest latency data injection to hadoop for quant analytics?

lowest latency data injection to hadoop for quant analytics?

I’ve customer looking for low latency data injection to hadoop . Customer wants to inject 1million records per/sec. Can someone guide me which tools or technology can be used for this kind of data injection to hadoop.

 

==

There is a number of solutions for loading data into HDFS: Flume, Scribe, Chukwa. Some teams load data into HBase as fast storage. If data is loaded from a relational database there is a Sqoop.

As usual the devil is in details. What is the size of a record? What are the latency requirements (msec, seconds, minutes)? How many sources of data? Is it a continuous data stream or a batch load?

 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Mellanox Announces World Lowest Latency for High-Frequency Trading


Mellanox Announces World Lowest Latency for High-Frequency Trading

Mellanox’s messaging acceleration software (VMA 6.0) delivers record-breaking performance for socket-based applications, including those used in…

http://www.hftreview.com/pg/newsfeeds/mellanox/item/32491/mellanox-announces-world-lowest-latency-for-highfrequency-trading

An off the shelf IP core and low-cost Xilinx setup already beats their “world record”.

 

would like to know more

 

Maybe STACResearch should more to develop benchmarking in this area

http://www.stacresearch.com/node/10807

 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

LMAX has a unique Java disruptor for multithreading latency for quant development. Now C++ and C# versions exists

LMAX has a unique Java disruptor for multithreading latency for quant development. Now C++ and C# versions exists

Here the links you should know about this:

http://stackoverflow.com/questions/6943569/lmaxs-disruptor-pattern-is-there-a-port-to-c

http://www.2robots.com/2011/08/13/a-c-disruptor/

From LMAX In Java:

http://code.google.com/p/disruptor/

C++ version:

http://code.google.com/p/disruptor-cpp/

 

http://www.infoq.com/presentations/LMAX

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Quant opinion: What is your latency on CME futures orders?

Quant opinion: What is your latency on CME futures orders?

I’m trading through an API provider (similar to TT X_Trader but someone else), and getting around 11ms between issuing an order and it appearing in the market. Personally I feel this is very slow, as I have a direct CME connection. What latency are other people able to get down to?

time between issuing an order and it appearing in the market should be easily less than 2ms.

 

–0

My last order took about 9 ms.

Should I yell at someone?

 

We see 4 – 8 ms but CME will not even discuss the range of internal latency. Does anyone know even a ballpark latency for the CME internal processing?

We are trying to determine the latency in different areas of our system and without the CME latency benchmark it is difficult to know what we are shooting for (lowest possible latency exclusive of CME).

 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Quant opinion Latency: Not All Numbers Are Measured The Same

Quant opinion Latency: Not All Numbers Are Measured The Same

Understanding how to measure Latency on Ethernet Switches

—-

What are the methods that you apply to get these measurements?

—-

This is a great subject, although to be correct it’s probably advisable to restate it as: Not all Parameters are measured the same way, depending on the context of intent. Think of all the variables that might affect latency on an instantaneous (vs. time-averaged) basis.

—-

Hey I didn’t create the title. There is also an almost identical document available on Cisco’s CCO pages.

@Afrim, yes that is a very good point and I would like to put that one out to the vendors on this forum. Any comments guys?

 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Quant development: The Latency Challenge

Quant development: The Latency Challenge

Only 13% attributed to the network and 65% to the application

advancedtrading.com

Strategies for using latency as a competitive advantage.

—-

As more players jump onto the HFT bandwagon, the use of exchange co-location, low latency switches, and hardware acceleration cards will no longer give you the edge. The edge is and always will be the application and how effectively it can use operating system resource. This is the tough stuff folks. To accomplish this, low latency team development teams should have embedded performance engineers or systems administrators with low level internals skills.

—-

It´s not even micro seconds anymore. Most of the big guys think in nano seconds! They´d be sitting with their machines right in the Exchanges´s server room if they could. What´s next? NO latency at all? Future, here we come.

—-

Huh… how many computers do you know of that can accurately track anything below microseconds?

—-

Some of our machines do in fact sit in the Exchange’s server rooms.

—–

I asked how many computers can track anything into the microseconds as straw-man type response.

Most all systems can if they have a reliable source of the time data they use and an evidence practice as well. Also the key issue is not to rely on the local TIME OF DAY (ToD) service to accurately track anything but rather to log to a controlled system with its time managed properly making the actual ToD in the production framework almost irrelevant.

We – Certichron happen to provide access to reference clock services which NIST itself operates. This is much better than a GPS service and provides 50ps or coarser access to the actual UTC standard itself. It also suffers none of the spoofability or other attacks GPS does and well – its about evidence of time.

How we do this is we cloned the US Timebase in Boulder and operate a fractional-instance of the timebase in concert with NIST Time and Frequency. its part of a technology we had them design a decade ago to facilitate placing an official NIST UTC instance in Tokyo (in the Nomura Trading Building on Shinjuku there).

So that’s how we deal with putting sub-microsecond time service into the trading framework. The first of these master timing centers and the regional access points in NYC and Bridgewater NJ also make this a complete service solution.

As to why GPS is not good enough… Most people fail to understand or even allow the review of the evidence model GPS produces and its horrible. The L1 GPS System can be spoofed and jammed so easily it is 100% unreliable for all securities applications.

Why we think NIST UTC is so important is that aside from not being the legal US Timebase per 15 USC 271 and 15 USC 272, GPS is a horrible source of evidence of anything. But it’s also important to notice that the spoofability of the L1 system is so easy that the Joint Chiefs of Staff issued a quiet-memorandum banning the use of unencrypted L1 services for all Military applications in 1998.

That order issued by the DoD alone disqualified it for all use as a trust-anchor for commercial transaction processing – and its something that the GPS industry as a whole has fought to scrub out of the public awareness since it would mean a large portion of the 1.5 B marketspace per year as reported on the WWW.GPS.GOV website. If you need more proof check out the SPOOFING GPS google searches or GPS Jamming…

That said a reliable source of time which can be proven after the fact is key to all market data operations and reporting!

—–

evidence is so compelling. Its just funny that most of the people around today take being told their evidence models were designed to meet the wrong set of standards pretty hard. They take that they relied on their vendors and were lied to in their time management practices pretty hard too, but hey… it is what it is.

NIST UTC rules!

 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Quant development: Virtualization Increase overall latency in trade life cycle ?

Virtualization Increase overall latency in trade life cycle ?

One study in US shows that If you implement Virtualization on Trading server It increase Latency up to as 44 microsecond (87 microsecond round-trip )

—-

Virtualization, in computing, is the creation of a virtual (rather than actual) version of something, such as a hardware platform, operating system. When you create something out of something latency would come.

—-

Virtualization is not to be used in an environment where time factor is highly critical. Virtualization is required for optimal usage of IT resources.

—-

everyone in this group know the basics of virtualization But No one know INSIDE Story.My Discussion is all about Inside as i allready mention for each & every implementation under Virtualization cause 87 microsecond delay I am 100% sure most of them out their do not know this fact.So please Do Not Mind but give some inside views ONLY……….

I had given the real reason of delay. While you had given the data. For four and half years I have worked as software engineer in NetApp “The storage Company” , NetApp is a partner of Vmware for development of Virtualized storage. I know exactly why the delay is there and why Virtualization in current form should not be implemented in mission critical appilications. I was just trying to keep it simple. In present form Every study and testing of virtualization will generate the delay data. If anybody is interested in specific technical reason, then please let me know.

Just NO virtualization on time-critical infrastructure. Bare-bone hardware servers with plenty of extra RAM/CPU/Bandwidth to cope with any activity peak. Everything else just runs find on VMs.

 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!