Tag Archives: considerations

Considerations for building a custom algo trading server

Questions from the same previous person last week

Hi Bryan.

 

I was wondering if you figured out what you were going to do about your server(s) situation?

 

You talked about getting some kind of cloud solution, on top of a hardware server.
I think you mentioned things like: redundancy and failover on linux Ubuntu.

 

You also said you probably would not be able to find things like this at Canada Computers…

So where would you go and what are the economics associated? Could this be done for under a $1000? What hardware and software options would be needed?

A colleague at work said I could buy a raid card to put into a computer with the proper software and cooling system so it could run 24/7 as a server.

What are  your thoughts?

 

Ok. Fair enough.

 

I guess this is not a decision to be made over a short period of time.
…lots to think about I guess.

 

I have worked in graphics departments that use a ton of data, so the first thing that came to mind was a raid server.

The only problem is that I don’t know if this is an option at and whether I need a very specific solution.

I made a video for all these answers
http://www.deltaserverstore.com/dell-server.html

https://distrowatch.com/

https://www.freebsd.org/

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

HFT architecture considerations with Fastflow in C++

HFT architecture considerations with Fastflow in C++


I just made a video on the considerations for a new open source architecture of a high-speed system for Interactive Brokers. The highlights to take away from this video include:

Possibly similar performances of an FPGA or GPU board thanks to this software accelerator feature

How Goldman Sachs uses an internal risk management database called secDB.

Check out this detailed video here

Hardware considerations for server with C++ Fastflow architecture

A few months ago I posted a potential server that could be great to use this new architecture on a bare-bones Linux system. I have listed what I think are the important features for any server that you would need for a high-speed trading system. 

Check out that list here.

Meetup: Message queue demos with NOSQL Redis C++ Java and Interactive Brokers

As a reminder, I will be doing demos on my chosen NOSQL in-memory fast database with Interactive Brokers TWS. This video will not be having any public access after the live session on Monday night.

Go here to get the details
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Hardware considerations for server with C++ Fastflow architecture

Hardware considerations for server with C++ Fastflow architecture

Key is throughput of your concurrent processing of the chips and kind as well

[7:48:26 PM] Bryan Downing: here is what I focus on

[7:48:40 PM] Bryan Downing: i am architecting the system:

[7:48:47 PM] Bryan Downing: as i am a mere mortal with a fixed budget

[7:49:15 PM] Bryan Downing: this system needs SSD (small amount is ok as it will only need some console apps with basic Linux services running)

[7:49:32 PM] Bryan Downing: boatloads of ram to house the data

[7:49:42 PM] Bryan Downing: processor is key but this is where you pay

[7:50:12 PM] Bryan Downing: 24 threads are needed for max throughput but for now, i can get a decent little system for under 1k

[7:50:23 PM] Bryan Downing: i can get a good cluster for $1500 as well

[7:50:36 PM] Bryan Downing: just for testing but as they system gets more sophisticated, i scale up the hardware

[7:50:51 PM] Bryan Downing: they key is to remove as many unnecessary services the OS will run

[7:51:25 PM] Bryan Downing: it will have less competition for the processor cycles which the my systems’s threading will compete for so it will be useless to run on windows

Example: https://quantlabs.net/blog/2014/12/meet-the-ibm-baby-beast-server-for-potential-high-speed-automated-trading/

 

This vendor is decent for anyone in Toronto http://canadaserver.wix.com/canadaserver#!ibm-servers/czn3

Join my FREE newsletter to learn what kind of hardware I get for this high speed trading system 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

HFT architecture considerations with Fastflow in C++

HFT architecture considerations with Fastflow in C++

View the video to see my logic instead of potentially bad typing

NOTE: In my video, I do address FPGA/GPU like performance through software accelerator. I also address the secret sauce tricks of Goldman Sachs system wide secDB high performance risk management.

My notes from Fastflow tutorial with Redis

Download from the package at http://sourceforge.net/projects/mc-fastflow/

This is from the fftutorial.pdf

P14 for node management

Figure 3.3 shows farms with feedback (collection) not W is worker, E is emitter while C is collector

Input stream pg 20 (hello_farm2.cpp) i.e. stage 1 for input from Redis, stage 2 for algo, stage 3 for trading decision

Or pg 21 with emitter and collector defined

No collector with the main memory or send them to the next stage (in case the farm is

in a pipeline stage) provided that the next stage is de_ned as ff_minode (i.e.

multi-input node).  Pg 22 hello_farm4.cpp

 

3.5 Feedback channels p 27

 

3.6 Mixing farms pipelines and feedbacks

FastFlow pipeline, task-farm skeletons and the feedback pattern modi_er can be

nested and combined in many di_erent ways. Figure 3.4 sketches some of the

possible combinations that can be realised in a easy way.

 

**** 3.7 Software accelerators like an FPGA

Using FastFlow accelerator mode is not that di_erent from using FastFlow

to write an application only using skeletons (see Fig. 3.5). The skeletons must

be started as a software accelerator, and tasks have to be o_oaded from the

main program. A simple program using the FastFlow accelerator mode is shown

below: see pg 30 accelerator.cpp

Could use img_farm+pipe.cpp or img_pipe+farm.cpp from figure 3.6

 

On pg 40:

The next step is to reduce the number of resources used. For example the farm

Emitter can be used to read _les from the disk, whereas the farm Collector for

writing _les to the disk. Furthermore, the blur and emboss _lters may be computed

sequentially using a single workers. This is the so called “normal form”

obtained optimising the resource usage. img farm . cpp
*** For fastest processing focus on those patterns that are stateless as map on pg 44 explains

Parallel_for maybe more powerful than how Matlab does it with more options

Pg 48 with ParallelForReduce shows how to use math routines like summary of array

ff_Map for FPGA like

pg 52 Why use the ff_Map instead of using directly a ParallelFor in a sequential

ff_node?

Pg 52 uses matrix multiplication matmul.cpp

P 43 mandel.cpp has image processing

P 57 sobel.cpp uses image

 

******* P60 ff_mdf uses graph instructions (just FYI: Goldman Sachs system wide on enterprise secDB works the same (hm………..) as in figure 5.1  à creating graph tasks on p61 hello_mdf.cpp

P63 block based matrix multiiplciation on could be used for complex matrix with linear algebra techniques (????)

 

Join my FREE newsletter to learn more about using Fastflow for potentially high speed trading systems 

 

 

 

 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!