Tag Archives: HPC

Translating HPC speak from NAG

Translating HPC speak from NAG

Ohh this is so good and true

http://top500.org/blog/translating-hpc-speak/

Join my FREE newsletter to learn more about high performance computing for automated trading

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Python RDMA ideal for HPC computing in HFT environment?

Python RDMA ideal for HPC computing in HFT environment?

RDMA is the unknown way to message between large blocks of memory like in gigabyte sizes

Note support for Mac OSX:

The SMB Direct Protocol, or SMB over RDMA allows zero-copy networking. This enables data transfer between the network adapter and application memory without data buffering in the operating system. However, I have not been able to find any evidence that this is supported in OS X.

OS X Yosemite (Server 4.03) and SMB3

Join my FREE newsletter if I plan to use this package in my automated trading

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Don’t be fooled by those expensive servers for HPC HFT computing from Oracle Sparc or Cray Youtube video shows its the programming

Don’t be fooled by those expensive servers for high performance HFT computing from Oracle Sparc, Cray, etc. This Youtube video show its the programming

I just sent this to someone:
It seems you are wanting to spend money on some high end servers. My experience with CUDA is it can only be used for backtesting, not real time analysis trading. Also, these expensive servers are just that, expensive. The performance techniques  all done in your programming:

You will only get this performance in C++ or .NET, Java is quite limited to achieve this.
I use Matlab Simulink to generate code either into C, C++, or FPGA ready if need be. You don’t need to blow your budget on expensive hardware like you listed.
I am learning this from one of the best quants in London UK. I talk with him for many hours per week. He does not hold back on what he knows.
Hope this helps

 

Learn more from my FREE newsletter

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

For HFT: Nvidia GPU CUDA super computing capability thanks to Tesla Compute Cluster with specific Windows HPC support

For HFT:  Nvidia GPU CUDA super computing capability thanks to Tesla Compute Cluster with specific Windows HPC  support

It looks like you can extend into buildnig super computing capability with Tesla Compute Cluster. It seems more geared to towards with Windows with HPC support but there is pribably painful Linux support. This enables you to chain multiple Tesla boards together so that is a big neat.

http.developer.nvidia.com/ParallelNsight/2.1/Documentation/UserGuide/HTML/Content/Tesla_Compute_Cluster.htm

http://www.nvidia.ca/object/software-for-tesla-products.html

See how I plan to to use this technology by joining my free newsletter 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

How to build Beowulf cluster aks HPC for Open MPI C++ apps like HFT with R with RCPP RInside package on Debian Linux #rstats #linux

How to build Beowulf cluster aks HPC for Open MPI C++ apps like HFT with R with RCPP RInside  package on Debian Linux #rstats #linux

Install Debian Etch from: and follow first CD install instructions:

http://www.debian.org/CD/http-ftp/

Install Debian Virtual Machines in Virtual Box as usual

http://www.wired.com/geekdad/2012/02/debian-linux-on-oracle-virtualbox/

Go with all default options.  You can always clone  a second  client VM to create a slave.

Set up hostname and configure network via: (choose the static address  not DHCP)

http://www.debian-administration.org/articles/254

Note that you could loose network connectivity soo add an additional Network Adapter with Bridged Adapter for each VM. within the Virtual Box setting. so see next step

Ensure each VM has network adapter setting to Virtual Box Host Only

http://christophermaier.name/blog/2010/09/01/host-only-networking-with-virtualbox

Create NFS for both virtual machines as server and client:

http://www.howtoforge.com/nfs-server-and-client-debian-etch

Note: When  editing exports on the server with:

/home           192.168.0.101(rw,sync,no_root_squash)

/var/nfs        192.168.0.101(rw,sync)

 

Note that 101 is the client!. ALso, I have when I try to auto mount during boot on the client, it would hang. I just created a shell script to do the mounts manually when you boot.  I just called this /etc/mountmaster.sh

 

If you need to add users do:

http://www.debian-administration.org/articles/2

You should be able to proceed with the following set up of your Beowulf cluster:

http://www.debian-administration.org/articles/2

To understand nad configure ssh connections with no passowrd, use:

SSH your Debian servers without password

This could be useful http://wiki.debian.org/SSH but note it uses RSA not DSA

If confused, use this next one but remember the ssh server is the Open MPI slave while client is the Open MPI since it logins into the Open MPI slave.

http://www.howtoforge.com/set-up-ssh-with-public-key-authentication-debian-etch

 

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Checking out Revolution R enterprise with RevoDeployR for Windows HPC high performance computing

Checking out Revolution R enterprise with RevoDeployR

I am going to check this out over the next few days. This was all due because of one package these guys have,. The scaling capabilities looks really good as they also use RevoDeployR for scaling within WIndows HPC. Remember that one? We played with it a long while ago.

UPDATE: Not a good impression when even the install package for Windows fails. I had problems with some of the product from this company. Confidence is not well, not there.

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Potential risk management data intensive at bank with: C++, C#, STL, Condor, Symphony, Gemstone Gemfire, Grid Computing for HPC needs

Potential risk management data intensive at bank with: C++, C#, STL, Condor, Symphony, Gemstone Gemfire, Grid Computing for HPC needs

From a recruiter sent:  (Please don’t ask who it was or where this posting is from), I am just highlighting some skill needs you may want to think about

The reason for m email is that my client, a tier 1 investment bank, is on the lookout for a highly technical C++ developer to work within the front office trade capture desk.

This position will be responsible for analysis, design and development of trading/risk system components, encompassing front-to-back functionality for various parts of front-office business areas. Development will be undertaken predominantly using C++ Windows, STL and some C#.

This is a permanent opportunity, based in London.

In this role, technical expertise of C++ windows and some C# understanding are both extremely important. (Though they are also considering developers from outside of banking as well!). You will be working extensively with front office developers and quants and learning from some of the industries finest!

This is a technically exciting position as you will be designing and developing trading applications using C++ and c# technologies. The ideal candidate will have worked strong C++ windows skills and some working knowledge of C#. Business knowledge is not required for this is a role which you will learn Grid Computing, there is also a high emphasis on vendor products for example Platform Symphony, Gemstone GemFire and Condor.

To be considered for this position you must have:

– C++ Windows, STL and COM
– C# (Advantageous)
– Grid computing or vendor products knowledge.

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Instead of HPC getting easier , we are now confronted with increasing complexity and a new trend…

Instead of HPC getting easier (as we dreamed of many years ago), we are now confronted with increasing complexity and a new trend…

… emerging almost every year, such as multicore and manycore, scaling up and out, big data, digital manufacturing and the missing middle, green computing, and HPC in the Cloud. For many, especially the end-user, this is a very painful and growing mixture of technical, mental, and even political challenges which no-one is able to handle individually anymore.

HPC is getting crowded

isc-events.com

If HPC is moving towards into the cloud there is – at least to me – the question what data is to be calculated there and how to transmit this data into the cloud ?
Is it generated by web accesses on a lot of clients each connected with the cloud data base, or is it locally within the cloud on the HPC system(s) generated, or transferred over local internet connection into the cloud ?

 

]IMHO, it’s a problem of the right wrapping of really complex technologies. iPod’s story as an example might be annoying, but, well, Apple won because they gave minimal required functionality to their customers, and took away any need to think about formats, codecs, bitrates, tags and that stuff. Since HPC is getting easier, it becomes more and more important to simplify an access to HPC applications (not to clusters or HPC infrastructure or even to set of nodes) for typical everyday users of such apps – engineers, researchers, to anybody else who doesn’t have even imagination about administrative aspects behind HPC applications.

 

==

The pendulum swings both ways. In technology that typically includes cycles of new hardware and software, the later always trying to address the failings of the former.

HPC will never get easier in one sense: as the computers become more powerful, the definition of a difficult problem will include previously-considered-impossible problems

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

100KW racks – ready for HPC for quant development?

100KW racks – ready for HPC for quant development?

We have just put together our first 20KW chassis (5 per rack). It’s warm refrigerant cooled (>30C) can handle 130W+ CPUs running continuously at full throttle and uses PCI express as the interconnect offering hardware latencies about 50% that of Infiniband.

We’d like to invite a few of you over for our open house on June 26th to give feedback. You’ll also have a chance to speak to representatives of several of the companies whose leading edge technologies have been incorporated in the system.

Email me or call 408 327 8100 to get time and location.

 

Rack cooling systems

clusteredsystems.com

 

Interesting: What extent of cooling power is possible with such solution ? 200kW, 250kW? or more ?

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

HPC – A Competitive Advantage for HFT

HPC – A Competitive Advantage for HFT

HPC Solutions – A competitive advantage by helping companies lower costs, boost productivity and improve efficieny http://www.appro.com/industry-news/hpc-also-a-competitive-advantage-expert-says/

==
“HPC is also a competitive advantage as more companies adopt the technology”

You know, working in an HPC group, and having written an HPC book, I am sometimes completely at a loss to understand what other people mean by “HPC”.

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!