Adventures in python and algorithmic trading

Sharky

Sharky

Staff
Messages
5,946
Likes
611
I thought it was time I threw my hat in the ring and start my own trading journal. So, before I change my mind, here it is!

In the coming weeks and months I'll be sharing my journey into an area of trading that particularly interests me. I'm not sure where this will end up, but I think it's going to be a fun ride... :p

RESOURCES LIST

I'm reserving this section of the first post to list out useful resources I come across on my travels. I'll keep it updated as and when I post updates, so you can always refer back here for a summary of all the links mentioned.

Learning Python
Python Algorithmic Trading
Udemy Courses (Recommended)
Options Software & Resources
Interactive Brokers
Non-Trading Resources (but still super useful!)
Reading List
Authors
 
Last edited:
Sharky - Have you started or are you going to be documenting as you go along your journey?
Awaiting in anticipation!
 
Thanks for the interest @iq200, so actually I started this week and plan to post an update at least weekly, probably on Sunday nights.
 
No, at least not in any meaningful way. I've pretty much read a few tutorials and looked over the syntax, that's about it. So anyone who's not a complete technophobe should, in theory, be able to follow along with me.
 
Having taught myself Python there are loads of free tutorials on the net in addition to many paid-for books covering different specialisms and all levels of complexity. If you want a no-nonsense intro book then I can recommend this (this particular publisher also does lots of other good stuff) for its simplicity and no frills.

PS - if you are completely new to programming then you may find it useful to study an introductory tutorial that covers the basics common to all programming languages (there are just a few basic constructs just as in any spoken language). I would also suggest that to go down the DIY programming route is great fun and of great interest but unless you have a geeky side to your personality and just enjoy doing it for its own sake, it is not a particularly time/cost-effective way of doing things. For most non-experts I would think that hiring in a competent programmer would be the most efficient way to proceed. After all – we did start out wanting to drain the swamp (ie get trading!) didn't we?

Python.JPG
 
Last edited:
This should be an interesting thread for me as I also have an interest in Python and as others have said there are some very good resources available FOC for learning it. I do have a question for everyone which is:

What IDE do people suggest to use for Python and why ?
 
Trader333 said:
This should be an interesting thread for me as I also have an interest in Python and as others have said there are some very good resources available FOC for learning it. I do have a question for everyone which is:

What IDE do people suggest to use for Python and why ?
Click to expand...

Python's own IDE "IDLE" is very good - does all that's needed and it's included in the free installation. But I also like another free and very simple IDE - "Thonny". Ideal for beginners, does auto indenting and uses colour effectively. It does code step-throughs and highlights syntax errors – all useful stuff for beginners and experts alike! It also has the advantage that when installed it also includes Python itself. https://thonny.org/ (it's interesting that this comes from Estonia – I've used several other pieces of excellent software from that country and it would seem they're pretty good at it).

My approach to programming & software in general (and trading) is to keep it as simple as possible and that includes the IDE – if you need something a lot more sophisticated and complex later on, well – leave that until later on.
 
Last edited:
Do you intend to discuss a specific book chapter by chapter or perhaps a course ?
Preferably free.
 
Will watch this thread with interest.

Keen to use an automated tool for intraday stuff, but having researched many forums etc, unsure where to start - Python/Matlab/C#,C++ - and how reliable a tool that I have built will be, running an IB account.

Python seems to be the best language for a newbie (to programming) to start with, but would prefer to pay a professional programmer to code my strategy with functioning error-handling etc and show me how to maintain/manage it in future.
 
0007 said:
Having taught myself Python there are loads of free tutorials on the net in addition to many paid-for books covering different specialisms and all levels of complexity... If you want a no-nonsense intro book then I can recommend this (this particular publisher also does lots of other good stuff) for its simplicity and no frills.

PS - if you are completely new to programming then you may find it useful to study an introductory tutorial that covers the basics common to all programming languages (there are just a few basic constructs just as in any spoken language). I would also suggest that to go down the DIY programming route is great fun and of great interest but unless you have a geeky side to your personality and just enjoy doing it for its own sake, it is not a particularly time/cost-effective way of doing things. For most non-experts I would think that hiring in a competent programmer would be the most efficient way to proceed. After all – we did start out wanting to drain the swamp (ie get trading!) didn't we?
Click to expand...

Thanks for the input, it's just the sort of commentary I was hoping to get from those with more experience in the topics covered. It'll be great for my own learning too. I'll be sharing my own findings too on the resources I've found useful, many of which are free. I'd agree that it helps massively to be a bit of a geek, then it doesn't feel so much like work, but more like solving a puzzle!

dentist007 said:
Hi Paul
Have you defined what you are looking for with your set up
Click to expand...

At the stage no, but I'm drawn towards options trading.

Trader333 said:
What IDE do people suggest to use for Python and why ?
Click to expand...

As @0007 points out Python offers IDLE out of the box, I haven't really spent time yet on evaluating the options. But at this stage, it sounds very much like Jupyter (iPython) notebooks are ideal for experimenting with strategy ideas. For my actual development environment I'll probably stick with my text editor of choice Sublime Text 3 (on Mac) which I've grown to like a lot whilst working on the new version of T2W.

Pat494 said:
Do you intend to discuss a specific book chapter by chapter or perhaps a course ?
Preferably free.
Click to expand...

I'll certainly be sharing lists of resources that I've found, many of which are free; and commenting on what I've found useful; this may well include chapters from books. As you might imagine there's no single book or resource that is going to tell you everything you need to know, so it's a good thing, I like to embrace the chaos of figuring out myself.

sideways-sid said:
Will watch this thread with interest.

Keen to use an automated tool for intraday stuff, but having researched many forums etc, unsure where to start - Python/Matlab/C#,C++ - and how reliable a tool that I have built will be, running an IB account.

Python seems to be the best language for a newbie (to programming) to start with, but would prefer to pay a professional programmer to code my strategy with functioning error-handling etc and show me how to maintain/manage it in future.
Click to expand...

I definitely think you'll find this interesting as I'll be trading with an IB account. First paper trading, then a real account. Python is indeed a very popular choice right now, but I can definitely see the attraction in getting someone to do this all for you. It's just there are so many options available, even if you're not going to code it yourself, I think having an awareness of those options is going to be super helpful.
 
First off, thanks for reading my journal, feel free to post your questions and comments any time. I'm especially keen to hear from the least and most experienced amongst you. The -least- because hopefully these adventures might actually be of some practical use to you, and the -most- because I have a lot to learn! Before we begin, I suggest you read the Risk Warning here: https://www.trade2win.com/help/risk-warning/ then re-read it. Algorithmic trading is not going to be right for everyone, and this isn't going to be painting by numbers. Please do you own research too!

A few things to note...

  • I've never blogged before, well apart from a road trip in California which lasted about 2 weeks, so I'm guessing it's going to take a little while to get in my stride. I've always been keen to do it though, and I finally think I've got a topic that might just be worth blogging about!
  • Although I'm tech savyy, I'm far from being a programmer. I don't have a degree in computing or mathematics. Any knowledge I have has been self-taught, I used to work (full-time paid) as a developer, but that was almost 20 years ago; and even then I wasn't that good. I consider myself, at best, a hacker, in the literal sense of hacking together something that just about works.
  • I'm time poor. I'm a husband, father (or a toddler) and two companies to run. So I've set a goal of an hour a day on this, which might be ambitious, but at least for now, seems manageable to fit in amongst everything else. Like compounded interest, those hours add up and if I can keep it up for a month, then I've got a great chance of establishing a habit and keeping it going.
  • Keeping the motivation going will be the hardest part, so in order to put the odds in my favour, there are a few things to do. Firstly, I've already done it by sharing this journal. Secondly, I need to have a goal in mind. Just getting into something without solid goals is a recipe for going nowhere. So here it is:

My goal is by the end of 2019, to having up and running, 100% automated trading of options, using Python and Interactive Brokers on a live account.

Why Options? Why Python? Why Interactive Brokers? I hear you ask.

Well, re. Options, I could blame it all on @Trading Spaces at the T2W Christmas Drinks for planting that seed in my head, but I dig the flexibility that options trading give you in profiting from the market regardless of direction, and let's face it who wouldn't want to uncover the secrets of iron condors, butterfly spreads and the wonders of the greeks.

Python, seems to be the poster child right now for algorithmic trading. I really like the Pythonic approach to programming, where the language itself gets out of the way as far as possible. It's a general language, so I can apply the knowledge I gain to other areas - great! It's hugely popular for machine learning and AI; and with a growing body of knowledge around Python for Algorithmic trading, it just seems like the obvious choice.

Finally, Interactive Brokers, they now have a native Python API, the fees are minimal, they offer paper-trading and they've scrapped their minimum account requirements. What's there not to like!

As I mentioned in an earlier post my plan is to post an update at least once a week, documenting my progress. I started out last week, and wanted to hit the ground running. So the first task I set myself was getting Python set up on my local machine (ie. the one I'm typing on now) and get it talking to Interactive Brokers. I wanted to be able to visualise how a DIY approach to automated trading works in practice and now I've already got a pretty good idea. Since I've run out of time for today, I'll post a follow up detailing what I achieved last week, in the next few days...
 
Okay, let's try and get up to speed with the little progress made to date (it's only start of week 2 after all).

First off, let's embrace the chaos. If you're starting out as I am, then it's somewhat daunting. You start googling around and before you know it, you've got more tabs open in chrome than your, supposedly capable, laptop can handle.

As I see it there's a bunch of areas I'll need to get proficient at, or at least get a handle on to make progress. Time is limited so I need to be smart about how I spend my time. Whilst I could easily choose to work through, say, an online python course over a month or two, it's just not targeted enough for my liking. At the same time, I'm not going to stress about trying to figure out the perfect use of my time either, for fear of paralysis by analysis, so I'm just going to be mindful about it and try and take each week at a time.

So, for someone who's new to this, a good place to start is understanding at a very high level how automated trading is going to work. I'll summarise them as follows.

The core components:
  • Python (language, libraries and tools)
  • Broker API (paper and live trading)
  • Server infrastructure (keeping a strategy running 24/7)
In other words, the nuts and bolts, a computer running my algorithm 24/7, using Python to communicate with Interactive Broker, through their API, receiving a data feed (most probably end of day, but could be intraday) and making decisions to buy/sell based on that data and sending those orders to the broker.

Then there's the actual strategy (or strategies) themselves which also covers a number of areas:
  • Options strategy (market selection/position sizing/money and risk management)
  • Prototyping (starting with an idea, and seeing if it's got legs)
  • Strategy backtesting & analysis (if it does then time to dig in)
  • Shiny Stuff (machine learning/AI/data sets)
That's it! Become an expert on those 7 bullet points, and you can call yourself an algorithmic options trader.

Or can you? Let's find out...

Week 1

So, for the first week I wanted to hit the ground running, and focus on the core components. I wanted to be able to visualise how it all works. Hence my first task was to get my computer talking to Interactive Brokers. I broke this down into the following:

1. Get python installed on my computer

Not a bad place to start, the official website: https://www.python.org/ and their documentation: https://docs.python.org/3/.

I'm on Mac, and I wanted to keep it simple so just set up a local development environment (no fancy cloud servers just yet).

I always like to get a second opinion on things, so a bit of googling on setting up Python and I found a handy tutorial: https://wsvincent.com/install-python3-mac/ (quick date check - 2018 - so should still be useful today). If you read that article and feel totally lost (and you're on Mac) then don't despair. You just break things down until you understand them, for me that means lots of googling, it might not work for everyone, but if I read 2 or 3 sources on the same topic, something normally sticks. I also have the benefit of having a familiarity with tools like Homebrew and knowing my way around the command line so that really helps!

2. Get my head around Interactive Brokers (IB) API

I opened an IB account a year or two ago (https://www.interactivebrokers.co.uk/en/index.php?f=4695), funded it but ended up not using it and closed it last year. Turned out that re-opening it was easy and just required a call to IB's customer support team. If you don't have an account, they do provide a test account that you can use with their API, but I've read the prices are 'total garbage' (unconfirmed) so you're better off applying for a real account and then using the paper trading (simulated account) that comes with it instead.

Obviously I'll want to test whatever strategies first on a paper trading account before risking my capital. Time to take note there are limitations to using it to be aware of:

https://www.interactivebrokers.com/en/software/am/am/manageaccount/paper_trading_limitations.htm
https://www.interactivebrokers.com/en/software/am/am/manageaccount/aboutpapertradingaccounts.htm

Trading permissions, market data subscriptions, base currency, and other account configurations are the same as specified for your regular account. The paper trading account start with $1m in GBP equivalent (since I'm in the UK) but this can easily be reset from the Account Management page of their website.

3. Moving on to the API itself..

The main page for Interactive Brokers API is here:
https://www.interactivebrokers.com/en/index.php?f=5041

IB offers a bunch of APIs in order to build automated trading applications, these include Java (popular), C++, DDE (used for Excel) amongst others. But we already know that Python is the one we're interested in.

Some reading around and IB offers tons of articles/videos in their knowledge base here:
https://ibkr.info/article/2484 (official guide to installing their API)
https://ibkr.info/index?title=&type=article (all articles)
https://ibkr.info/video/1034 (all videos)
https://interactivebrokers.github.io/tws-api/ (API reference guide)

As I mentioned I always like to jump on multiple sources when learning, and I came across this very readable blog post to installing the Python API https://qoppac.blogspot.com/2017/03/interactive-brokers-native-python-api.html

In order to get API working, it needs connectivity either via installation and running of Trader Workstation (TWS), which is IB's full blown trading application, or IB Gateway (IBGW), a lighter application that consumes about 40% fewer resources and only really useful if you're using the API, otherwise you can't do anything else with it.

TWS and IBGW are identical from the point of view of TWS’s API. Both represent a server to which an API client application can open a socket connection after the user has (manually) authenticated. For security reasons a headless session of TWS or IBGW without a GUI is not supported (but as with everything in life there are workarounds!)

Following the blog post above I got everything working, I tried it on TWS and then IBGW (note you can't have both running at the same time). My computer was talking to IB's server, for now, asking what time it was by running the code here, and receiving back something along the lines of..

Getting the time from the server... 1547591957

We're some way away from automated trading, but as a proof of concept, it worked.
 
Hi Paul...

ive always used quant / programming mates to do my grunt work ....so huge respect here to you

This allows me the objectivity of focusing on the trading system and not getting lost in the programming detail .....

will watch with interest ........

(dont let the programming get in front of the all important trading strategy)

as a complete Knob on EA programming ....can i ask why python is better than MT4 EA's .....which is where all my projects to date have been ?

N
 
I see you are using a Mac - I use the regular PC
and IB - I use MT4
Looks like we are diverging ??
 
I am going to be doing something similar this year if I have the time. Unfortunately, what I am undertaking isn't as simple as running some python code. I plan on automating a macro based system which will be monitoring economic data and news feeds and trading off of it. The code will require data streaming, natural language processing, AI with deep learning, and a complex trade management process. I will be doing this in azure because a single pc won't be enough to handle the processing and trade decisions. If Ray Dalio can do it why not me.
 
You must log in or register to reply here.

Similar threads

C
Yfinance
Replies
0
Views
1K
cassiopeia
C
H
Proposal - Open Source Trading SW - Open AI Desk
Replies
0
Views
2K
hANSIc99
H
McQuant
QuantConnect LEAN platform
Replies
2
Views
3K
frankhu
frankhu
S
Automated trading using MATLAB?
Replies
1
Views
3K
raghu
R
B
Python and Interactive Brokers
Replies
5
Views
3K
TickCOM
T
Top