What are the key characteristics of a ‘Big Data’ problem?

(Last Updated On: March 28, 2012)

What are the key characteristics of a ‘Big Data’ problem?

When can you say a problem can not be solved using tradition RDBMS and or BI tools.



I don’t think there is any straightforward answer to it. It all depends what is that you are trying to solve. Most of the typical scenarios do involve marrying the traditional, structured data stored in your RDBMS or data marts with the unstructured or semi structured data stored in logs or NoSQL databases. But if I have to answer your question then it will be mainly the data which is getting generated in bulk..GB/TB of data every day/week and you need to analyse that to get some insights into it to either use it for competitive differentiation or to define new business



BIG Data as such is not a problem. It is in fact solution to many unsolved problems if you know how to process in a cost effective and timely manner. Volume, velocity and variety are three key characteristics of BIG Data, not the Big data problem though. Traditional RDMBS have their own strengths, but it may be difficult to convert millions of scanned copies of news papers into PDF using RDBMS (just a use case example). Moreover, you may not need all the source data to be persisted but only the final aggregated output so another challenge you have there is high velocity streaming data. There can be many other examples and good use cases.



Thanks to both Partha and Vishal. The community have solved these kind of problems before using clustering , partioning and various other combos of tools. So is this a new buzzword for distributed computing? I am just trying to probe it a bit deeper and see if you gurus have any solid cases.



NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!
This entry was posted in Quant Analytics, Quant Development and tagged , , on by .

About caustic

Hi i there My name is Bryan Downing. I am part of a company called QuantLabs.Net This is specifically a company with a high profile blog about technology, trading, financial, investment, quant, etc. It posts things on how to do job interviews with large companies like Morgan Stanley, Bloomberg, Citibank, and IBM. It also posts different unique tips and tricks on Java, C++, or C programming. It posts about different techniques in learning about Matlab and building models or strategies. There is a lot here if you are into venturing into the financial world like quant or technical analysis. It also discusses the future generation of trading and programming Specialties: C++, Java, C#, Matlab, quant, models, strategies, technical analysis, linux, windows P.S. I have been known to be the worst typist. Do not be offended by it as I like to bang stuff out and put priorty of what I do over typing. Maybe one day I can get a full time copy editor to help out. Do note I prefer videos as they are much easier to produce so check out my many video at youtube.com/quantlabs