Tag Archives: hbase

Many companies like OPower are listing Hadoop Hbase and Hive as their big data base platform. Should we follow for quant based market tick data?

Many companies like OPower are listing Hadoop Hbase and Hive as their big data base platform. Should we follow for quant based market tick data?

Makes me wonder if we should get caught up in all the buzzie good stuff

Join our FREE newsletter to see we get caught up in it for big data analysis

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Which RDMS or NOSQL database do you use for R? MySQL, Cassandra, HBase, MongoDB, Oracle, PostgreSQL, CouchDB, SQLite?

Which RDMS or NOSQL database do you use for R? MySQL, Cassandra,  HBase, MongoDB, Oracle, PostgreSQL, CouchDB, SQLite?

Join my FREE newsletter to say what gets posted about R !

This R survey is kind of important. It will show a few things:

  1. Which R most users use regardless if they are commercial vs open source vs NOSQL .
  2. This will help us figure out which database is best for R using the scalability and speed depending on the requirements. This includes multiple writes for market tick data from C++ or a Java application and access by various R algorithms for analytics purposes.

Go here for the poll.

Here are some reasonable options with reasons:

MYSQL

I would assume this to be the number one choice since it is open source (or at least they say). It also contains sharding and other scalability needs with clustering. Is this something that people are using for their trading platform requirements? This includes using MYSQL as a tick data repository.

PostgreSQL

Is anyone actually using this open source database for their R needs?

Oracle

This is easily the most popular commercial RDMS for both Linux/Unix and Windows. As Oracle has open R into with a connector into their ecosystem, I wondered if people are actually using this.

SQL Server/DB2/Sybase

I am unsure if there are any R package connectors to any of these databases. I was just curious as I am really not interested in these as a real option.

Cassandra

There seems to be no R package support for this. I once posted something on R-Bloggers.com and it lit up the site, it made me wonder if this is actually more popular than people think. It seems to meet the needs of both quick write and read access.

Redis

Now the doRedis R package looked really hot. It even showcased how to use with a potential financial analytics system. I even saw Java sharding examples which left me excited on the capabilities of this database.

MongoDB

This seems to be strangely the most popular of all. I also found various R packages which seems to support it as well.

HBase which is part of Hadoop

Eh. No support even according to Revolutionary Analytics which their lacking install R package guides. I gave up pretty quickly on these R packages.

All others database options seem fine but the ones listed above seem the most viable for any R user as repository for scaling and clustering.

Go here for the poll.

http://quantlabs.net/surveys/2012/06/19/what-rdms-or-nosql-database-should-a-r-user-focus-on/

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Active Quant uses Hadoop and HBase. Cannot use because of no support for NOSQL Redis but will use another .NET HFT trading platform

Active Quant uses Hadoop and HBase. Cannot use because of no support for NOSQL Redis but will use another .NET HFT trading platform

Wlell this sort of sucks but is a blessing disguise. I cannot use Active Quant with mith Redis NOSQL database option I plan to go with. As a result, I am going back to my original HFT platform built in MS .NET and C#. I just need to test these C# Redis client apps. I am sure I will have an update.

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Active Quant for HFT with Hadoop, Hbase, Java, R but with Redis NOSQL for now

Active Quant for HFT with Hadoop, Hbase, Java, R but with Redis NOSQL for now

Join my membership here at:

http://quantlabs.net/membership.htm

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Active Quant is ready as an HFT trading platform . How to connect to Redis instead of Hbase subproject to Hadoop?

 

Active Quant is ready as an HFT trading platform . How to connect to Redis instead of Hbase subproject to Hadoop?

Active Quant has come a long way! I wish there was a Redis connection.

As I have switched to open source technologies including Java, I have narrowed down my search to 2 trading platforms to go with

Active Quant and Marketcetera

I have followed both for years but it seems there have been no updates to Marketcetera in years so I guess that rules out that puppy.

Active Quant has always interested me but as there were no easy to install guides, it looks like there has been movement on this. There is a Youtube video on how to install it. Also, there seems to be a cleaned up section on the new version which includes Hadoop and HBase. I have just moved my environment into R and Redis which seems to be a smart choice. Sadly, connected R into HBase can be tough through the RHBase as I could not get it working. Maybe I should make another attempt but I am very happy with the doRedis R package for this. The question is how to get around this challenge with Active Quant. Here is the link:

 

http://aq2o.activequant.org/

 

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

No to MYSQL: How to use Cassandra with R and RCasandra versus Hadoop and HBase

No to MYSQL: How to use Cassandra with R and RCasandra versus Hadoop and HBase

MYSQL looked good but can get pricey. Cassandra is true free open source and Twitter uses it!

http://blog.milford.io/2010/06/installing-apache-cassandra-on-centos/

HBase vs Cassandra: why we moved

Learn more on how I proceed with this with R as part of my high frequency trading:

http://quantlabs.net/membership.htm

Lucky me, I get a strange issue with RCassandra, Hmmm…probably a stupid thing but please comment if anyone has any ideas. Thanks

This could be a potential workaround but let me try for another day:

Big Analytics with R, Cassandra, and Hive

 

 

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Quant development: ActiveQuant open source project now does HBASE and Hadoop for time series storage

Quant development: ActiveQuant open source project now does HBASE and Hadoop for time series storage

A comment in my Linked In group from Ulrich the founder:

 

you could also have a look at AQ2o’s server, we use hbase and hadoop for time series storage.

The uberjar runs out of the box for getting started with development.

http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&discussionID=101112879&gid=3427378&commentID=72803560&goback=%2Egmr_3427378&trk=NUS_DISC_Q-ncuc_mr#commentID_72803560

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Watch Matlab do parallel computing on a GPU very easily and quickly? Why use R, Hadoop, MapReduce, HBASE, etc?

Watch Matlab do parallel computing on a GPU very easily and quickly? Why use R, Hadoop, MapReduce, HBASE, etc?

If you watched this webinar, you would be asking yourself several many questions:

  1. Why use R which seems kind of premature compared to Matlab?
  2. 2Even if you used this Revolutionary Analytics project with Hadoop, you still have so much to set up and manage compared to Matlab’s Parallel Toolbox.  This R project is no where complete as compared to the maturity of Matlab.

Regardless of cost, I still feel this makes you much more productive as compared to setting and up managing hardware and underlying technologies like Hadoop/HBASE/MapReduce etc with R. It just seems to me, it would not add up as compared to the productivity you get out box with Matlab. Sometimes the cost of time needs to be factored into something like open source technologies which are free. Just my worthless two cents.

Parallel Computing with MATLAB in Computational Finance

 

http://www.mathworks.com/company/events/webinars/wbnr51891.html?id=51891&p1=801727294&p2=801727312

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Quant development Addendum to Hypertable vs. HBase Performance Test (w/ mslab enabled)

Quant development Addendum to Hypertable vs. HBase Performance Test (w/ mslab enabled)

In our original test, we mistakenly ran the HBase test with the hbase.hregion.memstore.mslab.enabled property set to false. We re-ran the test with the hbase.hregion.memstore.mslab.enabled property set to true and have reported the results in this addendum to the test.

Synopsis: It slowed performance on the 10KB and 1KB tests and still failed the 100 byte and 10 byte tests with Concurrent mode failure

hypertable.com

Hypertable is an open source project based on published best practices and our own experience in solving large-scale data-intensive tasks. Our goal is nothing less than that Hypertable become the world’s most massively parallel…

==

I think that HBase community MUST prove now that you are wrong.

 

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Hypertable Beats HBase in Performance Test — HBase Overwhelmed by Garbage Collection

Hypertable Beats HBase in Performance Test — HBase Overwhelmed by Garbage Collection

Hypertable is an open source, scalable database, based on Google’s proprietery Bigtable design. It is similar to HBase except that it is written in C++ for optimal performance. In this High Scalability post, we summarize a test we recently conducted comparing the performance of Hypertable with that of HBase under a number of realistic workloads.

 

 

hmmm
Sounds like someone didn’t bother tuning HBase….
(I Kid, I Kid)

Its interesting, but someone self serving….

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!