3 hour video Q&A Meetup with London Quant on databases machine learning and loads of other tech talk from an awesome trader

NOTE TO ALL: It has been strongly recommneded to make these future session private for only paying members only.

Wow This was a 3 hour event where lots were anwered. We covered machine learning, database, lots of tech, and so on. Start learning from the best now!

It was said how we do surpass this one a it was easily the best yet!

Join my FREE newsletter to learn about other events online

Important links from the discussion:

http://discretelogics.com/teafiles/

http://www.amazon.ca/Handbook-Neuroevolution-Through-Erlang-Gene/dp/1461444624

http://www.nanex.net/aqck2/4680/20140929_10.06.48.000_1000ms_eGOOGL.0.gif

http://www.springer.com/computer/swe/book/978-1-4614-4462-6

 Here is the entire exchange thread of Meeutup:

Bryan Downing (to All – Entire Audience): hello everyone

Bryan Downing (to All – Entire Audience): hey all

Bryan Downing (to All – Entire Audience): how is everyone doing tonite or today?

John Ruiru Munge (to All – Entire Audience): Yes I can hear you

Bryan Downing (to All – Entire Audience): http://www.gnuplot.info/

Edwol (to All – Entire Audience): no

Edwol (to All – Entire Audience): I do some python, occasionally VB

Gigi (to All – Entire Audience): https://www.youtube.com/watch?v=hcRUqBm9GFI

John Ruiru Munge (to All – Entire Audience): thanks

Jeff (to All – Entire Audience): Hi all.

Jeff (to All – Entire Audience): Great how are you. Thanks

Ven (to All – Entire Audience): Hello Everyone

Geoff (to All – Entire Audience): Hello everyone

Edwol (to All – Entire Audience): Hi

Jeff (to All – Entire Audience): Sorry Gigi, can you please repeat what you said about the Interactive brokers connection?

George (to All – Entire Audience): Hello Everyone

George (to All – Entire Audience): What time is it inLondon?

Bryan Downing (to All – Entire Audience): hey

George (to All – Entire Audience): Can you give an introduction of London Quant?

Edwol (to All – Entire Audience): y

Geoff (to All – Entire Audience): I use Python

John Ruiru Munge (to All – Entire Audience): basic knowledge of python at the moment

Jeff (to All – Entire Audience): I use python

James Hirschorn (to All – Entire Audience): neve used it

Ven (to All – Entire Audience): Java . basic python

sysuser (to All – Entire Audience): I cant hear anything

sysuser (to All – Entire Audience): Is all ok ?

Edwol (to All – Entire Audience): y

John Ruiru Munge (to All – Entire Audience): yes

James Hirschorn (to All – Entire Audience): fine

George (to All – Entire Audience): hear you okay

Edwol (to All – Entire Audience): y

Jeff (to All – Entire Audience): n

George (to All – Entire Audience): yes

Sholom (to All – Entire Audience): yes

James Hirschorn (to All – Entire Audience): not yet

Geoff (to All – Entire Audience): yes

Ven (to All – Entire Audience): not yet

sysuser (to All – Entire Audience): I cant hear anything

Geoff (to All – Entire Audience): yes

John Ruiru Munge (to All – Entire Audience): N

Edwol (to All – Entire Audience): n

George (to All – Entire Audience): yes

sysuser (to All – Entire Audience): Please sugest if anyone hearing anything

Karen Tursunov (to All – Entire Audience): no

Ven (to All – Entire Audience): syuser- try connecting again.

George (to All – Entire Audience): no

Edwol (to All – Entire Audience): n

Karen Tursunov (to All – Entire Audience): no

Geoff (to All – Entire Audience): yes,

John Ruiru Munge (to All – Entire Audience): quantopian only

Sholom (to All – Entire Audience): no

John Ruiru Munge (to All – Entire Audience): what are your backgrounds

Shane (to All – Entire Audience): any automated system design that outperforms others? how frequently do you have to change strategies? what is the lifetime of a strategy?

Geoff (to All – Entire Audience): Check 12 strategies once a month

James Hirschorn (to All – Entire Audience): Gigi: Can you please type the name of the legit binary options broker? Thanks.

Gigi (to All – Entire Audience): Dukascopy, maybe IG

Geoff (to All – Entire Audience): optimize and walk forward every 30 minues

Shane (to All – Entire Audience): cool, thanks

Edwol (to All – Entire Audience): how much of a role if any does machine learning play in your systems ?

Jeff (to All – Entire Audience): Are your non-linear elastic timeframes related to fibonacci numbers?

Alex (to All – Entire Audience): They’re 10,000 a month starting with cheaper services, and up to 30k a month for bloomberg/reuters digitized news streams.

Steve Eberbach (to All – Entire Audience): I would call elastic time windows “event or conditionally closed- opened bars”. What events or conditions would you use to define time-elastic sampling intervals in streaming sequential data?

Shane (to All – Entire Audience): can you describe your systems at a high level?

Alex (to All – Entire Audience): up to 30k a month for bloomberg/reuters digitzed news

Alex (to All – Entire Audience): Are you using separate systems to process, backtest and analyze OLTP (in memory) and OLAP (historical data)?

Alex (to All – Entire Audience): database technology, in memory database, separate for historical data?

Alex (to All – Entire Audience): 2 different systems or just 1 for in memory?

Shane (to All – Entire Audience): can we get an intro /overview to start?

Alex (to All – Entire Audience): Could your system adapt to a 2008-level event, have you backtested drawdowns for that yr?

George (to All – Entire Audience): Automatic trading systems affect the order flow of the products it is trading. In principle, order flow change can effect how other traders trade around it. Is there a limit on how long one can can use an automatic trading system before it become ineffective? ie no longer generates a profit.

Alex (to All – Entire Audience): so you’re talking about mean-reverting strategies on the eurusd?

Alex (to All – Entire Audience): do you consolidate bar data from ticks or do do you make second or minute based queries to the database?

Alex (to All – Entire Audience): what do you think of newsql, or do you use your own nosql database with a c++ api?

Bryan Downing (to All – Entire Audience): check out teafiles

Shane (to All – Entire Audience): what is your average trade length in wall clock seconds? average price differential (pennies? ticks?

Bryan Downing (to All – Entire Audience): http://discretelogics.com/teafiles/

Sholom (to All – Entire Audience): How do you use Monte Carlo to test a non-parametric method?

Alex (to All – Entire Audience): what do you think about open-source in-memory databases used in conjunction with something like hbase/hadoop for historical data?

Alex (to All – Entire Audience): is your system lock-free for multiple threads?

Alex (to All – Entire Audience): so you don’t like matlab, r, or python for backtesting? You prefer .net for backtesting, and c++ for production?

George (to All – Entire Audience): Do you encript data your data?

Sholom (to All – Entire Audience): What do you use to estimate the current span of memory in the market (the set of possible market behaviors that are already factored into the price) ?

George (to All – Entire Audience): What role does volatility play in your trading system

Shane (to All – Entire Audience): how do you diivide your research time? improving technology implementation vs stategy development? what % time on each? or is there some other activity worth more time?

Alex (to All – Entire Audience): so it works like a mean-reverting seasonal/temporal strategy based on correlation optimized by a genetic algorithm?

Alex (to All – Entire Audience): you don’t like GA, you prefer decision trees and NNs?

Edwol (to All – Entire Audience): So you use SVM for patterns?

Alex (to All – Entire Audience): SOMs are technically NNs, though, no?

Alex (to All – Entire Audience): so you don’t like matlab, r, or python for backtesting? You prefer .net for backtesting, and c++ for production?

Shane (to All – Entire Audience): what is the biggest challenge you face now? improving models? liquidity? technology? other?

Sholom (to All – Entire Audience): when creating your own volatility function what are the basic input parameters you must consider?

Sholom (to All – Entire Audience): I propose the following: ?(? , ?1, ?2)

>>Where ? = S/A and A is the psychological price threshold level discovered using an empirical/non-parametric method.

>> ?1 = The price path to get to the current price S (specific price movement plus price velocity)

>> ?2 = The current span of memory in the market (the set of possible market behaviors that are already factored into the price)

Alex (to All – Entire Audience): what

Alex (to All – Entire Audience): the difference between GA and GP in your words?

Jeff (to All – Entire Audience): What is the minimum educational barrier for entry into HFT, do you need PhD in STEM related subject to succeed in HFT?

Steve Eberbach (to All – Entire Audience): On difference between genetic programming versus optimization: With genetic programming you do not consider the model-evolving program’s setting of the coefficients of terms as ” optimization” as you would when fitting optimim coefficients to terms for “pre-assumed equations” with genetic algorithm.?

Bryan Downing (to All – Entire Audience): https://bigml.com/

Shane (to All – Entire Audience): how often tdo you produce new models? what approach/process do you use to develop a model?

Edwol (to All – Entire Audience): do you specify the inputs like mutatiion % crossover, tournament size or is it random in your GP ?

Alex (to All – Entire Audience): related to edwol, so how do you deal with low-level selection, mutation, fitness, etc. levels?

Jeff (to All – Entire Audience): Have you tried combining evolutionary algorithms (GAs) with machine learning to find the optimum inputs to neural networks and/or any other model(s)?

Geoff (to All – Entire Audience): During the london day, how many periods are your non linear elastic periods broken into for the EUR

Alex (to All – Entire Audience): ga inside gp

Steve Eberbach (to All – Entire Audience): Have you tried the GP software called Discipulus?

Shane (to All – Entire Audience): I have to drop off, thanksvery much for your time

Alex (to All – Entire Audience): So GP just selects the proper GA from multiple simulated GAs?

Alex (to All – Entire Audience): basically us and asia time zones

Bryan Downing (to All – Entire Audience): any more questions?

Alex (to All – Entire Audience): GP has code generation?

Jeff (to All – Entire Audience): GP writes code in which languages? Highlevel or lowlevel like assembler?

George (to All – Entire Audience): I beleive that your system uses a some form of a machine learning strategy. Does your system make mistakes? If so, do you have a safeguards to prevent errors that could lead to losses?

Alex (to All – Entire Audience): so you coded your own custom GP library, there are many various software libraries that do GA, but I haven’t seen the GP you’re talking about.

Alex (to All – Entire Audience): so you’re just using windows/.net for gui passed onto windows using a messaging middleware, but it’s replacable, for instance you could also do web gui or android app with the middleware communication?

Bryan Downing (to All – Entire Audience): more questions ?

Jeff (to All – Entire Audience): Can I ask how profitable your systems are? (%)

Is your trading history public anywhere?

Edwol (to All – Entire Audience): How do you verify the veracity of your GP – eg confusion matrix, ROC …

Alex (to All – Entire Audience): but you continue to use windows for guis? what do you think about the qt library?

Edwol (to All – Entire Audience): great

Bryan Downing (to All – Entire Audience): more questions?

Bryan Downing (to All – Entire Audience): more comments?

Bryan Downing (to All – Entire Audience): wpf and xaml folks

Bryan Downing (to All – Entire Audience): more questions or comments ?

Jeff (to All – Entire Audience): Can HFT systems be designed for GNULinux?

Alex (to All – Entire Audience): what do you think of religious .net believers or for programming languages in general? .net/windows still is limited by the os vs. linux

Bryan Downing (to All – Entire Audience): any other questions or comments?

Alex (to All – Entire Audience): so why do you hate the jvm?, mono is garbage for high-performance systems, I only see it being used for niche, code generation on android for instance, or for some video games.

Bryan Downing (to All – Entire Audience): questions from the newbies?

Bryan Downing (to All – Entire Audience): comments ?

Jeff (to All – Entire Audience): Newbie here: Your systems are deployed to Linux or Windows servers?

George (to All – Entire Audience): Is trading volume used as aan imput for system’s decision to make a trade

Alex (to All – Entire Audience): so what do you think of erlang’s approach to multhreading?

Alex (to All – Entire Audience): how decent is fx volume data from lmax? accurate?

Jeff (to All – Entire Audience): related to Alex’s question above, any Lisp or Clojure used?

Ven (to All – Entire Audience): what is the ping time between your system and exchange on average

Jeff (to All – Entire Audience): gr8 question Ven

Alex (to All – Entire Audience): so you go from from c++/qt to .net, but you ignore the middleman, java, why?

Alex (to All – Entire Audience): do you connect to muliple LPs through lmax, or do you have independence t connections?

Jeff (to All – Entire Audience): Are you co-located next to the exchange or in a datacenter like LD4

Alex (to All – Entire Audience): so do citi, jpm, etc. in addition to lmax?

Alex (to All – Entire Audience): do you use a prime of prime fx brokerage?

George (to All – Entire Audience): Does the majority of your system profit’scome from retail traders or instituanal traders

James Hirschorn (to All – Entire Audience): Have to go. Thanks!

Alex (to All – Entire Audience): are there any financial sites you frequent, besides your bloomberg terminal?

Ven (to All – Entire Audience): what part of the system do we need to fine tune improve the ping time – strategy code, broker api, hardware, network IO or everything

Alex (to All – Entire Audience): so no mnra or ssrn? You don’t buy the academia?

Jeff (to All – Entire Audience): where can I get the book?

George (to All – Entire Audience): Trading is a zero sum game. One makes profits from another traders losses. Are the majority your profits profrits from retail traders (eg Joe trader who day traders for a living) or from instituational traders (eg traders who trade for Banks,)?

Jeff (to All – Entire Audience): sorry, what’s the title again?

Alex (to All – Entire Audience): so you prefer direct engineering/machine learn journals to financial journals/papers?

Bryan Downing (to All – Entire Audience): http://www.amazon.ca/Handbook-Neuroevolution-Through-Erlang-Gene/dp/1461444624

Sholom (to All – Entire Audience): http://books.google.com/books?id=MgNmy8PIeyQC&printsec=frontcover#v=onepage&q&f=false

Jeff (to All – Entire Audience): sorry my phone call dropped.   But I hope the video of this will be posted.

Jeff (to All – Entire Audience): gr8 thanks Sholom

Daniel Howard (Siesta Key, FL) (to All – Entire Audience): http://uploaded.net/file/xh3dtyuf/Handbook.of.Neuroevolution.Through.Erlang.pdf

Jeff (to All – Entire Audience): gr8 Daniel, tnx

Sholom (to All – Entire Audience): https://github.com/CorticalComputer/Book_NeuroevolutionThroughErlang

Alex (to All – Entire Audience): what do you think of hedge funds, they seem to not outperform the market, like the efficient market theory would say. Would you say that’s because people who can make money through prop trading, do not need to lose ‘other people’s money’?

Alex (to All – Entire Audience): do you have any investment recommendations for long-term holdings? Do you hold index etfs, cds, etc?

George (to All – Entire Audience): London Quant: Thanks for your discussion.

Alex (to All – Entire Audience): so you’re a pure trader and otherwise in cash?

Daniel Howard (Siesta Key, FL) (to All – Entire Audience): Thanks, LQ

George (to All – Entire Audience): Thanks Bryan

Alex (to All – Entire Audience): for long-term holdings, otherwise short-term fx speculation

Geoff (to All – Entire Audience): Thank you very much from SF

Sholom (to All – Entire Audience): Thank you very much!

Edwol (to All – Entire Audience): Thanks LQ

Ven (to All – Entire Audience): Thanks London Quant. Great learning for me

Daniel Howard (Siesta Key, FL) (to All – Entire Audience): Night all

Jeff (to All – Entire Audience): Awesome meeting, thanks Bryan, thanks LondonQuant

Bryan Downing (to All – Entire Audience): no problems jeff

Alex (to All – Entire Audience): do you load data into memory for backtesting? How much memory do you have/recommend for a single node to handle an in-memory db?

Alex (to All – Entire Audience): do you use centos? base-metal no virtualization?

Jeff (to All – Entire Audience): can HFT methods be applied to crypto currency markets?   Do you believe HFT enter crypto currency trading in future?

Rahul (to All – Entire Audience): what kind of other systems do you use to form strategies

Jeff (to All – Entire Audience): Can Joe Trader enter HFT?

Jeff (to All – Entire Audience): btw: thank you Sholom for the github link

Sholom (to All – Entire Audience): You’re welcome Jeff.

Alex (to All – Entire Audience): so the way your db is designed, it can’t work with unstructured data, only time-series data?

Jeff (to All – Entire Audience): thanks Gigi

Rahul (to All – Entire Audience): R or matlab?

Alex (to All – Entire Audience): so you use python for scripting on linux for maintenance of packages, etc. but not for critical trading?

Sholom (to All – Entire Audience): yeah Python has tons of bugs

Alex (to All – Entire Audience): you deleted this: so the way your db is designed, it can’t work with unstructured data, only time-series data?

Bryan Downing (to Jeff): any more questions?

Bryan Downing (to All – Entire Audience): any more questions?

Bryan Downing (to All – Entire Audience): comments?

Alex (to All – Entire Audience): do you use kde on linux? what’s your preferred de?

Alex (to All – Entire Audience): so you build c++ on eclipse for your system or are you using c++/qt?

Jeff (to All – Entire Audience): Do you trade independently or for a bank/fund?

If independent, how do you structure your corp/company for tax reasons?

Alex (to All – Entire Audience): how much hair have you pulled out over c++ and dealing with pointers for your system?

Alex (to All – Entire Audience): do you have semi-manual execution, more grey-box than black box? watching bloomberg terminal for events, etc.?

Alex (to All – Entire Audience): so you have execution traders in addition to programmers/engineering experts?

Geoff (to All – Entire Audience): No

Alex (to All – Entire Audience): are you spaniard?

Jeff (to All – Entire Audience): no

Jeff (to All – Entire Audience): Alex: Israel or Italy

Bryan Downing (to All – Entire Audience): does background really matter?

Alex (to All – Entire Audience): Genuine question: you’re good at reading accents, jeff?

Jeff (to All – Entire Audience): Alex: no..   so so, not good, not bad

Jeff (to All – Entire Audience): depending on which language

Bryan Downing (to All – Entire Audience): let’s focus on trading folks

Geoff (to All – Entire Audience): I agree

Jeff (to All – Entire Audience): Can I follow your lifestyle on Istagram or Facebook?

Alex (to All – Entire Audience): do you use basic visualizations like line charts for pnl, etc. or do you use more advanged graphics like heatmaps, etc.?

Bryan Downing (to All – Entire Audience): lightning charts

Gigi (to All – Entire Audience): http://www.nanex.net/aqck2/4680/20140929_10.06.48.000_1000ms_eGOOGL.0.gif

Gigi (to All – Entire Audience): http://www.nanex.net/aqck2/4680/20140929_08.56.20.000_20000ms_eGOOGL.1.gif

Alex (to All – Entire Audience): so you’re a full-time in house shop for everything from machine learning libraries, to the database to charts to even custom linux distribution, how much man years would you estimate that it would take to build it?

Rahul (to All – Entire Audience): HFT can lead to market crash sometimes?

Jeff (to All – Entire Audience): What software management tools do you use, if any. SVN/Suubversion, Git, etc.?

Alex (to All – Entire Audience): what do you think of news sentiment and analysis?

Alex (to All – Entire Audience): but you watch bloomberg, isn’t that hypocritical?

Alex (to All – Entire Audience): terminal, vs. using the bloomberg feed

Rahul (to All – Entire Audience): how do you manage trades at events?

Rahul (to All – Entire Audience): dividend and merger

Alex (to All – Entire Audience): exactly, you’re not quantifying sentiment, you’re still using discretionary bloomberg terminal, not using the machine readable feed and performing sentiment analysis on it

Alex (to All – Entire Audience): what do you think of the libor scandal?, many institutions are now banning bloomberg chat use for their traders via compliance

Rahul (to All – Entire Audience): @ Bryan: Keep on sharing your knowledge

Jeff (to All – Entire Audience): How do you manage your concentration when developing/programming your systems, do you outsource any tasks or build solo?

Alex (to All – Entire Audience): are you colocating your own hardware at ny4?

Jeff (to All – Entire Audience): Do you use Agile or any other methodologies like scrum.

Do you unit test your code?

Alex (to All – Entire Audience): so are you using any x86 asm, or is it pure c++, so it could run on other archs eventually?

Alex (to All – Entire Audience): you prefer gcc or clang?

Alex (to All – Entire Audience): do you use sharpe ratios, information ratios, etc. or do you focus on pure pnl per trade?

Jeff (to All – Entire Audience): Can you apply scalability of the web/mobile development methodologies for financial markets / HFT?

Alex (to All – Entire Audience): so you’re focusing specifically on the market microstructure of the euro, and if the euro broke-up it wouldn’t work?

Alex (to All – Entire Audience): so it’s semi-directionary, and you have your own EMS?

Alex (to All – Entire Audience): discretionary

Alex (to All – Entire Audience): so the ai engine is more of an advisor agent?

Rahul (to All – Entire Audience): some good strategies that you would like to share?

Jeff (to All – Entire Audience): If you have children teenagers, are your children any of the rich kids of instagram showing lambos, ferraris, yachts, and private jets?

Alex (to All – Entire Audience): what do you think of mandelbrot’s theories?

Jeff (to All – Entire Audience): Can you use atronomical data from as inputs to your neural nets?

Jeff (to All – Entire Audience): 🙂

Bryan Downing (to All – Entire Audience): ha ha

Alex (to All – Entire Audience): what do you think of the hurst exponent or markov models, what do you use to detect regimes or is this embedded in the GP?

Rahul (to All – Entire Audience): what do you think about future of algo trading? do algo traders reallly make money all time? is it really effective?

Alex (to All – Entire Audience): do you use boost? Did you design your own fix engine?

Geoff (to All – Entire Audience): Thanks Bryan and London Quant from SF

Jeff (to All – Entire Audience): Awesome talks Bryan. thanks Bryan, Gigi, LondonQuant and everyone making this possible.

Rahul (to All – Entire Audience): Thanks to all

Sholom (to All – Entire Audience): Best one yet! Thanks LQ!

Edwol (to All – Entire Audience): This was awesome – thanks to all who participated

Alex (to All – Entire Audience): talk about together in another webinar after feedback exchange

Jeff (to All – Entire Audience): How long have these talks been going on?

Jeff (to All – Entire Audience): I mean the meetups

Jeff (to All – Entire Audience): Is it all online?   I neeed to catch up

Steve Eberbach (to All – Entire Audience): Bryan! This was a good pow wow. I had to take my kid to the doctor, and now I’m back! Hope you have recorded the audio of what I missed!

Geoff (to All – Entire Audience): Bye all

Jeff (to All – Entire Audience): Thanks, bye all.

Steve Eberbach (to All – Entire Audience): Bye!

Edwol (to All – Entire Audience): bye

Rahul (to All – Entire Audience): bye