Tag Archives: SQLite

Back to Postgresql as MongoDB and SQLite fail

Back to Postgresql as MongoDB and SQLite fail

After heavier testing, I kept running into SQLite ‘database is locked’ message all the time. It is really frustrating so I resorted back to MongoDB. When I tried reading in my database with pyMongo Python package, I kept getting no results. I would then reset tthe script, then it would work in a wierd way. It seems to be unreliable so I gave up there. This was the first time I cam across this issue. The latest hacking of MongoDB does not help either. I tried Cassandra but they proved to be unreliable. So here we are. Back to Postgres I go. At least it is more reliable.

Join my FREE newsletter to see which database option proves to be most reliable

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

SQLite make C++ and Python to algo trading easier

SQLite make C++ and Python to algo trading easier

I am working on SQLite with C++. I just did it with Python but both programming languages were easy to implement with SQLite. I have out my positions management through callback capabilities from the SQLite APIs done. You still get the benefits of SQL with in memory capabilities. The big drawback is you cannot cluster this over multiple systems but I am sure someone already developed a solution for that.

These techniques have greatly improved my time by reducing the complexity of communication. I hope to demonstrate a complete solution of a C++ process ‘program’ that can manage multiple open positions (market entry) with some basic ‘algo’ to close them. Once the market data conditions are satisfied, an a market exit will take place. All of these are executed through C++ and SQLite callback. More to come in coming days

Redis is fine but the C++ support for a working library on Mac OS  sucks big time

In terms of speed, it might be possible SQLite could outperform since Redis depends on TCPIP. SQLite can be embedded right into the program

 

http://www.hammerprinciple.com/databases/items/redis/sqlite

Join my FREE newsletter to learn more about this path of automated trading

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

SQLite demo in Cpp for market data entry exit

SQLite demo in Cpp for market data entry and exit potential

Happy holidays from me myself and I. This year has been probably one the best years since I launched this website called Quantlabs.net. After messing around on various trading strategies and ideas, involved with the math, learning Python, and most of all dropping Microsoft Windows. I can say 2017 is hopefully going to be the year that it all gets turned around. What I have planned really excites me but I will be keeping it to my chest as close as possible.

For instance, I just posted a somewhat exciting technical stack for handling my high-speed set of positions with eventual auto stop loss and soft targeting for fun. It does sound complicated but the database that I chose was purely by accident. This involves SQLite which may be the surprise of all databases. Just watch my video below to see what I mean.

Get all the gory technical porn in this 20 minute video

I have talked about removing my “Python Infrastructure Building Block” course from my Quant Elite membership.  Over the next few days, that will become a reality. This is no difference then the removal of my Interactive Brokers API workshop course last week. I hate being a Scrooge like this but it’s to prepare for the next level of my business for next year. I’m sure you get that if you continue reading over the next few weeks on my plans. I’m actually pretty excited about it after six years of so-called “research “.

Thanks Bryan

P.S. Are you part of my Facebook programming group? There is lots of action going on it where we can all learn about software development in this exciting field.
Join here https://www.facebook.com/groups/quantlabsnet/
Also stay up to date on my Facebook Page https://www.facebook.com/quantlabsnet/
Twitter: twitter.com/quantlabs
Lasly, I do post about 20-30 Bloomberg ‘video charts’ on my Instagram at https://www.instagram.com/quantlabsnet/

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

SQLite demo in Cpp for market data entry and exit potential

SQLite demo in Cpp for market data entry and exit potential

Here is my source code below. Just remember I assume you will enter into the market before running this code. Also, I explain how the exit works

Join my FREE newsletter to learn more about how this C++ could work for your automated trading

 

 

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Apple Swift 3 with SQLite MongoDB Redis NOSQL

Apple Swift 3 with SQLite MongoDB Redis NOSQL

As said, I get open source solutions can be a real pain. So let’s try more with NOSQL open source solutions. Remember I am using Swift 3 with Xcode which makes it tougher to work with older Swift code. It seems the Swift code converter is useless for us newbies. Remember this is experimental at this point for me. I would also like to add these tutorial also include exposing myself to newer technologies like Docker and REST protocol. Let’s get this done.

http://www.theappguruz.com/blog/use-sqlite-database-swiftMONG

 

SQLite Tutorial: Getting Started

View at Medium.com

MongoDB with REST

https://www.teskalabs.com/blog/tutorial_ch04_ios_rest_integration_mongodb

https://github.com/TeskaLabs/SeaCatTutorials/tree/master/ch03_ios_rest_integration/RESTClient <– What idiot would this monster of Hell?

https://github.com/swizzlr/swift-redis

 

Join my FREE newsletter to learn how these Swift solutions work out for me

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Demo of fetch and insert in Python with SQLite

Demo of fetch and insert in Python with SQLite

Another simple demo of this combo for an automated trading system

http://zetcode.com/db/sqlitepythontutorial/

Python and SQLite tutorial

Join my FREE newsletter to learn more about automated trading

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Python and SQLite tutorial

Python and SQLite tutorial

As it i looks like I am using SQLite as my preferred database (not my fave) to get working with C++ on a Mac OSX environment, I am doing this tutorial

http://zetcode.com/db/sqlitepythontutorial/

Join my FREE newsletter to leartrn how I plan to use SQLite with Python/C++ in my new automated trading system 

 

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Which RDMS or NOSQL database do you use for R? MySQL, Cassandra, HBase, MongoDB, Oracle, PostgreSQL, CouchDB, SQLite?

Which RDMS or NOSQL database do you use for R? MySQL, Cassandra,  HBase, MongoDB, Oracle, PostgreSQL, CouchDB, SQLite?

Join my FREE newsletter to say what gets posted about R !

This R survey is kind of important. It will show a few things:

  1. Which R most users use regardless if they are commercial vs open source vs NOSQL .
  2. This will help us figure out which database is best for R using the scalability and speed depending on the requirements. This includes multiple writes for market tick data from C++ or a Java application and access by various R algorithms for analytics purposes.

Go here for the poll.

Here are some reasonable options with reasons:

MYSQL

I would assume this to be the number one choice since it is open source (or at least they say). It also contains sharding and other scalability needs with clustering. Is this something that people are using for their trading platform requirements? This includes using MYSQL as a tick data repository.

PostgreSQL

Is anyone actually using this open source database for their R needs?

Oracle

This is easily the most popular commercial RDMS for both Linux/Unix and Windows. As Oracle has open R into with a connector into their ecosystem, I wondered if people are actually using this.

SQL Server/DB2/Sybase

I am unsure if there are any R package connectors to any of these databases. I was just curious as I am really not interested in these as a real option.

Cassandra

There seems to be no R package support for this. I once posted something on R-Bloggers.com and it lit up the site, it made me wonder if this is actually more popular than people think. It seems to meet the needs of both quick write and read access.

Redis

Now the doRedis R package looked really hot. It even showcased how to use with a potential financial analytics system. I even saw Java sharding examples which left me excited on the capabilities of this database.

MongoDB

This seems to be strangely the most popular of all. I also found various R packages which seems to support it as well.

HBase which is part of Hadoop

Eh. No support even according to Revolutionary Analytics which their lacking install R package guides. I gave up pretty quickly on these R packages.

All others database options seem fine but the ones listed above seem the most viable for any R user as repository for scaling and clustering.

Go here for the poll.

http://quantlabs.net/surveys/2012/06/19/what-rdms-or-nosql-database-should-a-r-user-focus-on/

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

Redis NOSQL proven for C++ in memory databases as compared to SQLite, Berkeley DB for HFT, Save $250K as compared to kdb +

Redis NOSQL proven for C++ in memory databases as compared to SQLite, Berkeley DB for HFT, Save $250K as compared to kdb +

Hi there

My march to building a high frequency trading is gathering steam pretty quickly. My backend called Redis has proven itself.

Finally, I got Redis working with a complete demo of its high speed in memory database capabilities. I even tested it with 100 millions rapid inserts . The video demo of this is here. All the C++ source code is available for my Premium members .

–> Join here to get access it <–

Do I really need to go on the benefits of this?  
As compared to other C++ in memory databases, SQLite is way to limiting as you can really only
have one client insert at a time. I find Berkeley DB is very cumbersome and
involved to set up. The tutorials I also saw were not really helpful. Thanks Oracle!
As for those posts on Monday, I decided to come away with a general consensus is
that most people within the community of QuantLabs.net are real world practioners
as the most popular postings referred back to cost or commercial vendors. Here are
some examples:
Now the way I am going, Redis alone could save me a $250K as compared to a typical kdb+ implementation.
If it proves itself in terms of speed and stability, this could save me thousands.
Redis could also save me the nightmare of maintaining horrific code if done in FPGA, CUDA, or
GPU. Bleh.
Did I mention the recent announcement of how R can handle 12000+ cores? Believe it. Until next time.
Remember, I have Youtube videos showing C++ calling R. Is this the next generation in HFT? 
Again, until next time.
See ya
Bryan
P.S.

Better get on that Premium Membership as I raise the rates. It may get pricey as I start posting these critical C++ components to help you build an speedy HFT platform from the ground up.

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!

No to Oracle TimesTen? It seems Redis is the next logical choice in a C++ HFT as compared to SQLite or BerkeleyDB

No to Oracle TimesTen? It seems Redis is the next logical choice in a C++ HFT as compared to SQLite or BerkeleyDB

It is always that debate of which is the best real time trading engine for C++ platform with R analytics through RCpp/Rinside.

First I mentioned Oracle TimesTen which look solid over the last weekend. It seems someone who is highly experienced has convinced me the product is entirely slow. Who knew? I am not in a position to argue so I will take their word for it. Without doubt, that lead me to three choices:

1. Berkeley DB

2. Redis

3. SQLLite

After reading the usual BerkeleyDB vs SQLite debate, it seems BDB wins without doubt as any trading platform needs concurrency which BDB offers over SQLite.

http://doc.gnu-darwin.org/intro/dbisnot.html

http://www.oracle.com/technetwork/database/berkeleydb/learnmore/bdbvssqlite-wp-186779.pdf

http://stackoverflow.com/questions/2824135/how-fast-is-berkeley-db-sql-compared-to-sqlite

As indicated above, BDB is not a network database or database server.

So I guess that brings me back to Redis. I can only use the hiredis C client which is recommended by their site. The C++ client is very outdated.

Some further decent comparisons with other NOSQL as well:

http://perfectmarket.com/blog/not_only_nosql_review_solution_evaluation_guide_chart

I would have gone with MongoDB but when you read this for C++,

http://www.mongodb.org/pages/viewpage.action?pageId=133415

My experience is the same with MongoDB:

I am dumping NOSQL including MongoDB clustering, trying MYSQL Cluster

http://news.ycombinator.com/item?id=2052852

http://stackoverflow.com/questions/2376846/which-key-value-store-is-the-most-promising-stable

http://vschart.com/compare/berkeley-db/vs/redis-database

Learn more what I do with these by being a Premium Member. Join here for instant access!

HOW DO YOU START A PROFITABLE TRADING BUSINESS? Read more NOW >>>

NOTE I now post my TRADING ALERTS into my personal FACEBOOK ACCOUNT and TWITTER. Don't worry as I don't post stupid cat videos or what I eat!