I was wondering if you figured out what you were going to do about your server(s) situation?
You talked about getting some kind of cloud solution, on top of a hardware server. I think you mentioned things like: redundancy and failover on linux Ubuntu.
You also said you probably would not be able to find things like this at Canada Computers…
So where would you go and what are the economics associated? Could this be done for under a $1000? What hardware and software options would be needed?
A colleague at work said I could buy a raid card to put into a computer with the proper software and cooling system so it could run 24/7 as a server.
What are your thoughts?
Ok. Fair enough.
I guess this is not a decision to be made over a short period of time. …lots to think about I guess.
I have worked in graphics departments that use a ton of data, so the first thing that came to mind was a raid server.
The only problem is that I don’t know if this is an option at and whether I need a very specific solution.
I made a video for all these answers
http://www.deltaserverstore.com/dell-server.html
I just made a video on the considerations for a new open source architecture of a high-speed system for Interactive Brokers. The highlights to take away from this video include:
Possibly similar performances of an FPGA or GPU board thanks to this software accelerator feature
How Goldman Sachs uses an internal risk management database called secDB.Check out this detailed video here
A few months ago I posted a potential server that could be great to use this new architecture on a bare-bones Linux system. I have listed what I think are the important features for any server that you would need for a high-speed trading system.
Check out that list here.
As a reminder, I will be doing demos on my chosen NOSQL in-memory fast database with Interactive Brokers TWS. This video will not be having any public access after the live session on Monday night.
Go here to get the details
Hardware considerations for server with C++ Fastflow architecture
Key is throughput of your concurrent processing of the chips and kind as well
[7:48:26 PM] Bryan Downing: here is what I focus on
[7:48:40 PM] Bryan Downing: i am architecting the system:
[7:48:47 PM] Bryan Downing: as i am a mere mortal with a fixed budget
[7:49:15 PM] Bryan Downing: this system needs SSD (small amount is ok as it will only need some console apps with basic Linux services running)
[7:49:32 PM] Bryan Downing: boatloads of ram to house the data
[7:49:42 PM] Bryan Downing: processor is key but this is where you pay
[7:50:12 PM] Bryan Downing: 24 threads are needed for max throughput but for now, i can get a decent little system for under 1k
[7:50:23 PM] Bryan Downing: i can get a good cluster for $1500 as well
[7:50:36 PM] Bryan Downing: just for testing but as they system gets more sophisticated, i scale up the hardware
[7:50:51 PM] Bryan Downing: they key is to remove as many unnecessary services the OS will run
[7:51:25 PM] Bryan Downing: it will have less competition for the processor cycles which the my systems’s threading will compete for so it will be useless to run on windows
HFT architecture considerations with Fastflow in C++
View the video to see my logic instead of potentially bad typing
NOTE: In my video, I do address FPGA/GPU like performance through software accelerator. I also address the secret sauce tricks of Goldman Sachs system wide secDB high performance risk management.
Figure 3.3 shows farms with feedback (collection) not W is worker, E is emitter while C is collector
Input stream pg 20 (hello_farm2.cpp) i.e. stage 1 for input from Redis, stage 2 for algo, stage 3 for trading decision
Or pg 21 with emitter and collector defined
No collector with the main memory or send them to the next stage (in case the farm is
in a pipeline stage) provided that the next stage is de_ned as ff_minode (i.e.
multi-input node). Pg 22 hello_farm4.cpp
3.5 Feedback channels p 27
3.6 Mixing farms pipelines and feedbacks
FastFlow pipeline, task-farm skeletons and the feedback pattern modi_er can be
nested and combined in many di_erent ways. Figure 3.4 sketches some of the
possible combinations that can be realised in a easy way.
**** 3.7 Software accelerators like an FPGA
Using FastFlow accelerator mode is not that di_erent from using FastFlow
to write an application only using skeletons (see Fig. 3.5). The skeletons must
be started as a software accelerator, and tasks have to be o_oaded from the
main program. A simple program using the FastFlow accelerator mode is shown
below: see pg 30 accelerator.cpp
Could use img_farm+pipe.cpp or img_pipe+farm.cpp from figure 3.6
On pg 40:
The next step is to reduce the number of resources used. For example the farm
Emitter can be used to read _les from the disk, whereas the farm Collector for
writing _les to the disk. Furthermore, the blur and emboss _lters may be computed
sequentially using a single workers. This is the so called “normal form”
obtained optimising the resource usage. img farm . cpp
*** For fastest processing focus on those patterns that are stateless as map on pg 44 explains
Parallel_for maybe more powerful than how Matlab does it with more options
Pg 48 with ParallelForReduce shows how to use math routines like summary of array
ff_Map for FPGA like
pg 52 Why use the ff_Map instead of using directly a ParallelFor in a sequential
ff_node?
Pg 52 uses matrix multiplication matmul.cpp
P 43 mandel.cpp has image processing
P 57 sobel.cpp uses image
******* P60 ff_mdf uses graph instructions (just FYI: Goldman Sachs system wide on enterprise secDB works the same (hm………..) as in figure 5.1 à creating graph tasks on p61 hello_mdf.cpp
P63 block based matrix multiiplciation on could be used for complex matrix with linear algebra techniques (????)
