Tag Archives: Fastflow

Notes on open source C++ Intel TBB vs Fastflow

Notes on open source C++ Intel TBB vs Fastflow

You should know about my Youtube video playlist of Fastflow. As I could not build it on my Mac OSX, I sent out a query to their email list. This was in reference to why their code would not build. Response was:

Your message for list fastflow@di.unito.it
has been rejected by the moderator (torquati@di.unipi.it).

(Subject of your mail: Mac OSX build errors)

Check fastflow list usage:
https://www.di.unito.it/listserver/info/fastflow

wtf? I could build Intel TBB on my Mac no problems with an event provided Xcode solution file. So guess what I will be working with as of now?  Here is what I dug up on this library:

https://www.threadingbuildingblocks.org/documentation

https://compscicenter.ru/media/slides/hp_course_2014_spring/2014_04_06_hp_course_2014_spring.pdf

https://goparallel.sourceforge.net/wp-content/uploads/2014/07/PUM18_Threading_Building_Blocks.pdf

http://www.cs.cmu.edu/afs/cs/academic/class/15499-s09/www/handouts/TBB-HPCC07.pdf

https://software.intel.com/en-us/node/506110

https://software.intel.com/en-us/node/506102

https://www.threadingbuildingblocks.org/tutorial-intel-tbb-concurrent-containers

https://www.threadingbuildingblocks.org/intel-tbb-tutorial

https://www.threadingbuildingblocks.org/docs/help/reference/task_scheduler/priorities.html

Simplest one: http://www.ibm.com/developerworks/aix/library/au-intelthreadbuilding/

https://software.intel.com/en-us/blogs/2011/09/09/a-feature-detection-example-using-the-intel-threading-building-blocks-flow-graph <– closest example for automated trading

http://www.drdobbs.com/tools/the-intel-threading-building-blocks-flow/231900177?pgno=1

https://software.intel.com/en-us/node/506216

Join my FREE newsletter to learn more about how Intel TBB can help in automated trading

 

Most useful for graph processing with events

Look at multifunction_node
For queueing look at queue_node and priority_queue_node

Five dining philosopher example

Streaming feature detector <—

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Will C++ Fastflow be as fast as GPU or FPGA hardware?

Will C++ Fastflow be as fast as GPU or FPGA hardware?

This is the closest I could get for a comparison

http://paraphrase-ict.eu/paperList/hpcc2012

Join my FREE newsletter to see how C++ help in high speed automated trading

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Fastflow multicore C++ for potential HFT like rapid trading

Fastflow multicore C++ for potential HFT like performance trading

Over the coming weeks, you’ll start to see me focus more on trading concepts versus technology. There seems to be a much more lucrative interest from professional traders recently so I’d like to start focusing on them in the future. I’ve already done an email on that a few days ago.

 

As I start to focus on more low latency, lower-level trading software to meet higher speeds of both market tick capturing and order management. I will not really call this high-frequency trading but this is being designed with that in mind. The biggest and most crucial aspect of it all is multi-core processing for all your algorithms analyzing done in parallel. There are a few languages that can do this but C++ is the most efficient which is why all the HFT shops to use it. We all know that is the way in terms of being a standard.

 

So if you go onto my YouTube channel, you will find a playlist that focuses on a C++ multi-core framework called FastFlow. Most recently, I just made a 20-minute video on the future of my high speed automated trading software.

 

 Check out this video here

The final dance? High speed trading software architecture

In fact, I have set aside Monday night to do a live webinar Meetup where you can challenge me on all these concepts. Any technical person is clearly invited to talk about it. Just to let you know, this will be the only time I will talk about it in detail. Also, don’t expect any source code to be released for it. I will only be presenting high level concepts in my usual ghetto presentation style

 

Get detail on this one time only event details here

Join one of my Meetups listed in the link above.

Pretty trading charts with Matplotlib and PyQtChart

The following night, for Tuesday, I will be presenting all the desired Python and Qt choices I have at my disposal. They are of course open source.

 

Get the details here

 

This is the second last module I am presenting as part of my “Independent trading business in Python” course series. The very last module which will be presented on Tuesday, May 3 where I will be discussing graphical user interface rapid development with Qt Designer.

I always have my source code available with live demos for question-and-answer’s at these times.

 

We are coming down to the wire before I start getting into the next phase for strategy development starting the following Tuesday on May 10. This first phase will focus on pair trading or arbitrage for any equity that Yahoo Finance tracks. That actually gives you access to over 158,000 tradeable instruments! Consider this: this is where pure maximum potential resides! Why trade link a chump? And in coming weeks to months, all of this will disappear. Vanish! Goodbye!

 

Once again, this is part of my Algorithmic Trading course series.

 

Get immediate access here

 

Or get full benefit description and details here

 

Thanks Bryan

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

HFT architecture considerations with Fastflow in C++

HFT architecture considerations with Fastflow in C++


I just made a video on the considerations for a new open source architecture of a high-speed system for Interactive Brokers. The highlights to take away from this video include:

Possibly similar performances of an FPGA or GPU board thanks to this software accelerator feature

How Goldman Sachs uses an internal risk management database called secDB.

Check out this detailed video here

Hardware considerations for server with C++ Fastflow architecture

A few months ago I posted a potential server that could be great to use this new architecture on a bare-bones Linux system. I have listed what I think are the important features for any server that you would need for a high-speed trading system. 

Check out that list here.

Meetup: Message queue demos with NOSQL Redis C++ Java and Interactive Brokers

As a reminder, I will be doing demos on my chosen NOSQL in-memory fast database with Interactive Brokers TWS. This video will not be having any public access after the live session on Monday night.

Go here to get the details
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Hardware considerations for server with C++ Fastflow architecture

Hardware considerations for server with C++ Fastflow architecture

Key is throughput of your concurrent processing of the chips and kind as well

[7:48:26 PM] Bryan Downing: here is what I focus on

[7:48:40 PM] Bryan Downing: i am architecting the system:

[7:48:47 PM] Bryan Downing: as i am a mere mortal with a fixed budget

[7:49:15 PM] Bryan Downing: this system needs SSD (small amount is ok as it will only need some console apps with basic Linux services running)

[7:49:32 PM] Bryan Downing: boatloads of ram to house the data

[7:49:42 PM] Bryan Downing: processor is key but this is where you pay

[7:50:12 PM] Bryan Downing: 24 threads are needed for max throughput but for now, i can get a decent little system for under 1k

[7:50:23 PM] Bryan Downing: i can get a good cluster for $1500 as well

[7:50:36 PM] Bryan Downing: just for testing but as they system gets more sophisticated, i scale up the hardware

[7:50:51 PM] Bryan Downing: they key is to remove as many unnecessary services the OS will run

[7:51:25 PM] Bryan Downing: it will have less competition for the processor cycles which the my systems’s threading will compete for so it will be useless to run on windows

Example: https://quantlabs.net/blog/2014/12/meet-the-ibm-baby-beast-server-for-potential-high-speed-automated-trading/

 

This vendor is decent for anyone in Toronto http://canadaserver.wix.com/canadaserver#!ibm-servers/czn3

Join my FREE newsletter to learn what kind of hardware I get for this high speed trading system 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

HFT architecture considerations with Fastflow in C++

HFT architecture considerations with Fastflow in C++

View the video to see my logic instead of potentially bad typing

NOTE: In my video, I do address FPGA/GPU like performance through software accelerator. I also address the secret sauce tricks of Goldman Sachs system wide secDB high performance risk management.

My notes from Fastflow tutorial with Redis

Download from the package at http://sourceforge.net/projects/mc-fastflow/

This is from the fftutorial.pdf

P14 for node management

Figure 3.3 shows farms with feedback (collection) not W is worker, E is emitter while C is collector

Input stream pg 20 (hello_farm2.cpp) i.e. stage 1 for input from Redis, stage 2 for algo, stage 3 for trading decision

Or pg 21 with emitter and collector defined

No collector with the main memory or send them to the next stage (in case the farm is

in a pipeline stage) provided that the next stage is de_ned as ff_minode (i.e.

multi-input node).  Pg 22 hello_farm4.cpp

 

3.5 Feedback channels p 27

 

3.6 Mixing farms pipelines and feedbacks

FastFlow pipeline, task-farm skeletons and the feedback pattern modi_er can be

nested and combined in many di_erent ways. Figure 3.4 sketches some of the

possible combinations that can be realised in a easy way.

 

**** 3.7 Software accelerators like an FPGA

Using FastFlow accelerator mode is not that di_erent from using FastFlow

to write an application only using skeletons (see Fig. 3.5). The skeletons must

be started as a software accelerator, and tasks have to be o_oaded from the

main program. A simple program using the FastFlow accelerator mode is shown

below: see pg 30 accelerator.cpp

Could use img_farm+pipe.cpp or img_pipe+farm.cpp from figure 3.6

 

On pg 40:

The next step is to reduce the number of resources used. For example the farm

Emitter can be used to read _les from the disk, whereas the farm Collector for

writing _les to the disk. Furthermore, the blur and emboss _lters may be computed

sequentially using a single workers. This is the so called “normal form”

obtained optimising the resource usage. img farm . cpp
*** For fastest processing focus on those patterns that are stateless as map on pg 44 explains

Parallel_for maybe more powerful than how Matlab does it with more options

Pg 48 with ParallelForReduce shows how to use math routines like summary of array

ff_Map for FPGA like

pg 52 Why use the ff_Map instead of using directly a ParallelFor in a sequential

ff_node?

Pg 52 uses matrix multiplication matmul.cpp

P 43 mandel.cpp has image processing

P 57 sobel.cpp uses image

 

******* P60 ff_mdf uses graph instructions (just FYI: Goldman Sachs system wide on enterprise secDB works the same (hm………..) as in figure 5.1  à creating graph tasks on p61 hello_mdf.cpp

P63 block based matrix multiiplciation on could be used for complex matrix with linear algebra techniques (????)

 

Join my FREE newsletter to learn more about using Fastflow for potentially high speed trading systems 

 

 

 

 

 

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Monoid design pattern to Fastflow multihreading C++ library for high speed trading like HFT

Monoid design pattern to Fastflow multihreading C++ library for high speed trading like HFT

As I started digging further into Monoid pattern thing, it was promising with these articles:

https://www.fpcomplete.com/blog/2012/09/functional-patterns-in-c

https://isocpp.org/blog/2013/05/functional-patterns-in-c-bartosz-milewski

https://www.quora.com/Why-is-monad-more-famous-than-functor-and-applicative-functor

One of these links does have source code demo. To be honest, the code was not commented or documented at all which means I cannot go beyond than looking at. It did not compile with my GCC 4.9 but I have no patience to figure it out from there.

Check out the history here:

C++ event driven meta programming libraries

Or watch the video here:

Event driven C++ Metaprograming

As these links base the methodology off Haskell, there are a number of reasons which this functional programming language could be as fast as C or C++. Here are some other comparisons:
http://lambda.jstolarek.com/2013/04/haskell-as-fast-as-c-a-case-study/

https://www.quora.com/Why-is-monad-more-famous-than-functor-and-applicative-functor

As I am no expert here, but it was recommended another multithreading library could handle the same performance using these Monoid design patterns. It does look promising but after some further digging, Intel TBB or Boost Futures came up. I think that is a yucky proposition when I knew about a faster (and easier) multi-threading library called Fastflow.
I took a look to see if it was abandoned. To my surprise, 2.1 just came out 2 days ago. Talk about perfect timing!

http://sourceforge.net/projects/mc-fastflow/files/?source=navbar

There was a performance graph to show how Fastflow performs against other libraries. It seems to keep up with OpenMP which is the fastest one compared to CILK or TBB. Again, I am no expert here but I think it is worthy to revisit this multithreading library.

fastflow 236876

It is also comforting to know the FIX8 project chose FastFlow as well for concurrency which they claim their library is quite fast. Knowing all this with my revisit to both C++ on Linux, it might important to showcase my earlier on this library:

https://quantlabs.net/blog/?s=fastflow

Videos: https://www.youtube.com/user/quantlabs/search?query=fastflow

Keep your eyes peeled on more up to date topics on Fastflow

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

TBB vs Fastflow for C++ parallel multithreading? Next steps for you

I have posted a detail series of next step I intend to take with deploying a live trading systematic model using code generation from Matlab Simulink to hopefully a fully automated trading C++ enabled system.
TBB vs Fastflow for C++ parallel multithreading? Next steps

 

Yes I have come down to the selection of C++ with Matlab to enable this deployment with Oanda as the chosen broker for simple forex trading!

Read more here 

TBB vs Fastflow for C++ parallel multithreading? Next steps

TBB vs Fastflow for C++ parallel multithreading? Next steps

Don’t forget about my next non trading techie talk. I like to open this up to retail traders:

Let’s talk non techie trading stuff MEETUP

Monday, May 11, 2015

7:00 PM

I got some non techie trading ideas I want to talk about. Let’s do that to share your investment ideas.

Let’s talk non techie trading stuff MEETUP

Login Details here:

1.  Please join my meeting, May 11, 2015 at 7:00 PM Eastern Daylight Time.

https://global.gotomeeting.com/join/450975573

2.  Use your microphone and speakers (VoIP) – a headset is recommended.  Or, call in using your telephone.

Dial +1 (224) 501-3212

Access Code: 450-975-573

Audio PIN: Shown after joining the meeting

Meeting ID: 450-975-573

GoToMeeting®

Online Meetings Made Easy®

Not at your computer? Click the link to join this meeting from your iPhone®, iPad®, Android® or Windows Phone® device via the GoToMeeting app.

Let’s end this email here so thanks for reading

Bryan

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Software accelerator for FPGA or GPU? Please Help out in building open source FastFlow C++ library for Windows

Software accelerator for FPGA or GPU? Please Help out in building open source FastFlow C++ library for Windows

I don’t ask for much but if you have some knowledge, let me know by posting here:
This would be most beneficial to everyone out there wanting to learn something from this project. It was mostly based around Linx or Mac OSX but I struggle with the Windows version.
NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Is FastFlow Open Source C++ library a fast software alternative to GPU or FPGA for HFT and quant?

Is FastFlow Open Source C++ library a fast software alternative to GPU or FPGA for HFT and quant?

I discovered this years ago but there have been some major enhancements since then. Here are some newer links:

http://www.1024cores.net/home/technologies/fastflow

http://calvados.di.unipi.it/dokuwiki/doku.php?id=ffnamespace:usermanual

Learn how we plan to apply this library in the future through our FREE newsletter

Here is an updated version of FastFlow 2.01 :  http://alpha.di.unito.it/parallel-programming-framework/

Do note! I cannot complete the builds so if anyone can let me know how to complete, that would be appreciated

As with any open source C__ project, I have decided to put this on the backburner. The instructions to build on Windows are inadequate so I don’t have the patience to try any further.

 

Read BUILD file but do make Visual Studio projects via:

cmake ..

Also could use forVisual Studio 2012:

>cmake -G”Visual Studio 11 Win64″ ..

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!