What are you using to load/stage your big data?

(Last Updated On: September 13, 2011)
Learn the Secret

Get  our 2 Free Books

Get these now which land directly to their inbox.
Invalid email address

What are you using to load/stage your big data?

I’m curious to know which software (messaging bus, etc) people are using to gather data and process it for their big data, low latency needs. Are there any newcomers in the group to threaten 29West (informatica) or Tibco’s (ftl) ?? How about AMQP based solutions ?? Or zeromq ?? Do products like flume have a roll here ??

Let’s hear about your experiences and software selection in this area.


In an earlier post you pointed out that there’s this ‘low-latency’ component to this group.
How much latency can you tolerate? If you’re not looking at low-latency, then flume should be the default.

If you’re looking at streaming data for ‘real time’ analysis. (or near real-time) that would be an interesting question. (And even if Hadoop/HBase would be a good fit in the first place.)



NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!
This entry was posted in Quant Development and tagged , , on by .

About caustic

Hi i there My name is Bryan Downing. I am part of a company called QuantLabs.Net This is specifically a company with a high profile blog about technology, trading, financial, investment, quant, etc. It posts things on how to do job interviews with large companies like Morgan Stanley, Bloomberg, Citibank, and IBM. It also posts different unique tips and tricks on Java, C++, or C programming. It posts about different techniques in learning about Matlab and building models or strategies. There is a lot here if you are into venturing into the financial world like quant or technical analysis. It also discusses the future generation of trading and programming Specialties: C++, Java, C#, Matlab, quant, models, strategies, technical analysis, linux, windows P.S. I have been known to be the worst typist. Do not be offended by it as I like to bang stuff out and put priorty of what I do over typing. Maybe one day I can get a full time copy editor to help out. Do note I prefer videos as they are much easier to produce so check out my many video at youtube.com/quantlabs