Adventures in python and algorithmic trading

[Sharky]

Nothing up and running as of today. I had to put things on hold towards the end of last year for personal and work reasons. But now I've somewhat cleared the decks, I'm going to be able to get back to it - still very much interested in python and systematic trading. I fully expect in the next 6-12 months to have something automated set up and running, then I'll get be able to answer that question

 

[Sharky]

Just a quick update, to say my current learning path is as follows:

• On Chapter 5 of Real Python Basics Book (https://realpython.com/products/python-basics-book/) - I bought this as part of a bundle for $30 three years ago and thought I'd better finally make use of it. So far it's pretty good, taking you through the real basics of python but ideal for building a strong foundation. Important to physically type in every exercise, you can't pick up programming without developing some physical memory muscle.
• I quickly read through the Trading Evolved book and now I've gone back chapter by chapter working through the examples. I finally got my first Zipline backtest successfully working, as in Chapter 7, via Anaconda and a Jupyter notebook as recommended in the book. I gave up trying to get it to work on my Mac, I'm running on my linux laptop instead.

I'm aiming for at least 1 hour a day on basic Python programming, and then another hour on the quantitive side (so right now, that's working through the Clenow book).

Since my previous updates, when I was getting rather excited about QuantConnect as a platform, I've updated my thinking somewhat. Quantopian is back to updating their open-source trading libarary Zipline, they've just released v.1.4 after a 2 year hiatus (https://github.com/quantopian/zipline/releases). Since it's used in Trading Evolved, as the backtesting engine of choice, I'll probably stick with it for now. As I mentioned Quantopian doesn't support live trading out of the box. But there are workarounds, and the most interesting looking one for me right now is Quantrocket, which is in active development and appears to let you plug-in Zipline as your backtesting engine, and trade live with either Interactive Brokers or Alpaca (which I think is only available to US customers). The downside to Quantrocket is that it doesn't come free, not unless you're prepared to stick to sample data, but a personal account trading, for example, up to a $1m account size is $199/month for 1 user, which seems reasonable if it potentially provides a complete turnkey solution to automating your trading (depending on your account size). Very early days, but at the moment this is the direction I'm moving towards.

 

[FXX]

I've moved to c# for my project with some java libraries but 95% c#. I will be providing a comprehensive update on my progress in due course. Looking forward to your trials and errors when you get stuck in.

 

[Pat494]

I wish you guys had opted for MT4 and then I could have contributed.

 

[FXX]

I wish you guys had opted for MT4 and then I could have contributed.

well I will need to figure out how the hell I can get my algo output into mt4 as an indicator. Any ideas how I can do that? I can build an api that provides the data but I have read that you can't build an indicator that makes an api call.

this is the issue technically

WebRequest - Common Functions - MQL4 Reference

WebRequest - Common Functions - MQL4 Reference

docs.mql4.com

 

[Pat494]

I guess by now you have ideas about the route you will take with systems and indicators.
Will you be using a grid system with martingale perhaps ?

 

[TickCOM]

Are you aware of what it takes in time and effort to perhaps end with something profitable ? It takes years, many years. Perhaps, because very few of those using a decade of their life reach any useful result.

Add to above, that any strategy will stop working when the market change behaviour, so the development time is longer than the strategy will work.

First one will need to pass the learning curve, involving try-and-error (edit-debug-compile-backtest) of 10-thousands of different types of strategies, and their variants, this learning curve cannot be bypassed.

 

[FXX]

Are you aware of what it takes in time and effort to perhaps end with something profitable ? It takes years, many years. Perhaps, because very few of those using a decade of their life reach any useful result.

Add to above, that any strategy will stop working when the market change behaviour, so the development time is longer than the strategy will work.

First one will need to pass the learning curve, involving try-and-error (edit-debug-compile-backtest) of 10-thousands of different types of strategies, and their variants, this learning curve cannot be bypassed.

it depends on many things. if you are just testing configurations of indicators and price then you could be doing it with mixed results for the rest of your life.

markets move for various reasons and its definitely not because a macd or any other indicator gave a signal.

 

[TickCOM]

it depends on many things. if you are just testing configurations of indicators and price then you could be doing it with mixed results for the rest of your life.

markets move for various reasons and its definitely not because a macd or any other indicator gave a signal.

Yes it does help to think,

 

[1nvest]

First one will need to pass the learning curve, involving try-and-error (edit-debug-compile-backtest) of 10-thousands of different types of strategies, and their variants, this learning curve cannot be bypassed.

go to quantpedia. there are hundreds of tried and tested strategies that are commonly known, to ones used by hedge funds, that you can use as a starting point. choose a few that meet your criteria (ex timeframe) do your own backtesting, add your own filters, use your own data set or instruments. it doesn't have to be perfect, as nothing is , but you can very very easily get a market beating strategy that could involve just one indicator, and two instruments

 

[TickCOM]

go to quantpedia. there are hundreds of tried and tested strategies that are commonly known, to ones used by hedge funds, that you can use as a starting point. choose a few that meet your criteria (ex timeframe) do your own backtesting, add your own filters, use your own data set or instruments. it doesn't have to be perfect, as nothing is , but you can very very easily get a market beating strategy that could involve just one indicator, and two instruments

One thing is ready made strategies, another is to develop your own.

As to ready-made you can also go to MyFxBook or sites who sell EA´s, many brokers have EA´s to use, some EA´s are even free to use. And we are back to the old days scam, pictures of a Ferrari next to a pool, bikinigirls, promising you get rich in no time. Today above has been replaced with stories about hedge funds in the attempt to lend credibility , where most hedge funds are in anyway not profitable.

This theme hasnt changed the past 15 years, the questions and answers are still same;

If you have a working strategy would you post it in public ?
What happens when 200 traders use the same signal ?
Would you store the code of a highly profitable strategy on an alien server ?
How long does it take till the strategy stop working - is it working right now ?
When was the strategy developed, and in which timeframe ?
Do you have sufficient data to backtest a strategy trading in the 30 minutes timeframe ?
Does it make any sense to backtest a strategy using data 3 years back ?
Why is it the longer back you optimize, the less trades per day you get ?
Which kind of strategies will you get if backtesting any strategy on any stock (hint long only due to 10 years bullish rally) ?
Is it possible to trade with profit one strategy, using data from one instrument ?

 

[1nvest]

If you have a working strategy would you post it in public ?
why not, that's why people have written books, its our due diligence in verifying this. but without a shadow of a doubt, working strategies are available on line and they still work now
What happens when 200 traders use the same signal ?
they will make money if it was a good signal. that's for the conspiracy theorists, thousands of mutual funds/hedge trades use the same "type" of strategies. they still work. quantpedia has those
Would you store the code of a highly profitable strategy on an alien server ?
why not, the IP is already out there, written in journals, papers, books. the alien server is nothing more than a repository
if i had come up with my own take on something, of course, if i could make money from it
How long does it take till the strategy stop working ?
only time will tell, its about making money in the now, it doesn't stop me using the strategy. i don't think my momentum strategy will stop working in the future, therefore i wont trade it now. market dynamics change, i may have to change my instrument if bond yields carry on, or if inflation goes up, i just have to adapt, but my core strategy will remain the same
When was the strategy developed, and in which timeframe ?
mine evolves constantly and is more on a weekly even monthly timeframe. im interested in longer term trends and making money for the long term. you need to choose what works for you, strategy is still the same though
Do you have sufficient data to backtest a strategy trading in the 30 minutes timeframe ?
get the data, data is available, you may have to pay, but that's hardly a stumbling block
Does it make any sense to backtest a strategy using data 3 years back ?
depends on the strategy, as long as you cover periods that are representative of market cycles
Why is it the longer back you optimize, the less trades per day you get ?
thats probably a result of curve fitting Which kind of strategies will you get if backtesting any strategy on any stock (hint long only due to 10 years bullish rally) ?
ones that work in bullish periods, obviously. thats why you cover different market cycles
Is it possible to trade with profit one strategy, using data from one instrument ?
Absolutely

my answers in red, its really not rocket science
you stated you needed to go through 10s of thousands of strategies..you dont, you need a starting point thats all
you dont need to reinvent the wheel
my answers will likely differ for others, we all have our own strategies, but i certainly didnt have to spend years and years when there is information already available to me, to you to anyone who just needs to look

 

[Pat494]

I like to test my strategies NOT by back testing them BUT by using it in a micro account on say 16 instruments ( the main ones like Eur/Usd etc.).I expect a weekly profit of at least 3% of my wedge.

 

[Pat494]

I guess by now you have ideas about the route you will take with systems and indicators.
Will you be using a grid system with martingale perhaps ?

bump bump bump
c;mon sharky share your thoughts mate.......................

 

[Sharky]

Thanks for the recent posts, I could share my thoughts but I don't think I've got much to add at this stage. My focus is definitely not get rich quick, what I'm attempting to do increase returns whilst reducing risk versus the 'market' by diversifying across asset classes and strategies - somewhat akin to building a mini hedge fund. I'm not shooting for astronomical returns, but if I can edge towards a 15% return a year, by taking an automated portfolio approach to investing that reduces risk and dreaded drawdowns then hip-hip-horrah! I don't plan to reinvent the wheel even, as 1invest pointed out QuantPedia is a great resource that's done most of the legwork for you in identifying tried and tested strategies. Along the way I hope to develop my python programming skills, better understand the whole algorithmic trading life cycle and finally get read and maybe put to good use a ton of trading and investing books I've picked up over the years.

Anyway, wanted to give a quick update, on progress in the last 3 weeks. So, it feels like it's been a bit all over the place, which probably reflects a) the fact that there's a ton of stuff to learn, b) there's no clear path to learn it, and c) we're all different, and we learn in different ways. Knowing myself I like to take a scattergun approach, which I need to be careful about, as I like to learn by touching multiple sources, but you can quickly become overwhelmed, so it's almost a psychological battle to feel like you're progressing at a rate, that keeps things interesting, makes you feel like you're progressing, whilst not quickly realising how much effort it's going to take.

I've come to the conclusion, that I only really need to know at this stage the basics of Python, and libraries like Numpy, Pandas, Matplotlib in order to feel comfortable reading algorithmic trading code. More important is choosing a path to the end goal which is live trading. Right now, I'm feeling good about using Zipline (open source Quantopian backtesting engine) running on QuantRocket on a cloud server, probably AWS. So my goal at the moment is to get my head around a few things 1) Quantopian/Zipline, meaning things like their Pipeline/Algorithm APIs primarily, as well as Pyfolio and Alphalens - these are all available on Zipline. To get off the ground I think it's helpful to start playing around with Quantopian, as you can immediately start coding and see the results, without worrying about configuring a local installation or getting the data you need. Then I plan to start playing around more Zipline to better understand the differences. 2) Trading Evolved book - my choice of Zipline has a lot to do with the fact that Clenow has chosen it for his book, so I plan to attempt to some of the strategies there up and running on Zipline myself, finally 3) QuantRocket - once I've got them running on my own local Zipline environment, I'll try moving these over to QuantRocket and attempt to live trade them with a paper trading account on IB (which I already have set up).

Before I got to this point, I was spending most of my time on Python learning, through a few sources:

• DataCamp
• SoloLearn
• Real Python ebook
• Various Printed Python Books I've accumulated over the years

I also started learning Flask (as a lightweight web framework) my thinking being to improve my Python I really need to get into a project that's going to interest me. Perhaps coding up a contest on T2W, as I know they're popular, but I quickly realised the time invested in that would right now be better served getting me nearer my immediate goals listed above, so give time is always a constraint, I'll put that on hold for now - but would live to come back to providing some tools or contests on T2W in due course.

 

[Sharky]

I'll try and post a weekly update going forward, but before the next big one let me just share a tip with you all. Check out the free courses on Kaggle here: https://www.kaggle.com/learn/overview - in particular Python then Pandas to get you started. Their method of combining lessons with interactive exercises works really well. I've found that unless you're coding when you're learning, it just doesn't stick. You should read to reinforce what you've learnt but there's something about muscle memory when you have to think and type things out, that can't be beaten.

 

[Sharky]

Another free resource I can recommend which I came across today is https://pythonprinciples.com/ - ideal if you're completely new to Python, they have lots of mini challenges to actually get you programming which is so important. It's not enough to simply read or watch videos, you need to give your fingers a workout too. If that's not challenging enough for you then check out the exhaustive list here: https://medium.com/datadriveninvestor/top-10-online-challenge-websites-on-python-3afe98ade09d

On a sidenote, I came across this article yesterday: https://www.oreilly.com/content/algorithmic-trading-in-less-than-100-lines-of-python-code/ - which promises to get you algorithmic trading in less than 100 lines of python code. Unfortunately I quickly found it's out of date, using an old Oanda API that no longer works. But I did manage to find somone who'd rewritten the article to use their new API here: https://github.com/benjaminchodroff/oandamomentum - I managed to get it working as far as pulling price data and plotting it on a graph, although it didn't seem to want to trade on my Oanda practice account and I didn't want to get too sidetracked getting any deeper into it. What I did learn however is that you can spread bet with Oanda and their API works for that too (not just CFDs and FX) which to me is attractive, given that here in the UK your profits are tax free - no need to declare anything on my self assessment.

 

[Sharky]

Another short update.

I've played around with a bunch of code challenge sites, and my favourites are: CodingGame, CodeWars, Exercism, Sololearn (mobile). If I had to narrow it further, right now I'd recommend CodeWars on the desktop and Sololearn ('Play' feature) on mobile. Both are free (the latter is ad supported). I definitely think if you spend an hour a day practising on these, within a couple of weeks, your basic coding and problem skills will really improve.

I also started this week the 'Applied Data Science with Python Specialization' offered by University of Michigan on Coursera. You can check it out here: https://www.coursera.org/specializations/data-science-python

 

[Sharky]

One more tip, came across the JetBrains Academy yesterday for learning Python:

JetBrains Academy | Learn Tech Skills With Developer Tools

Whether you're starting out in your tech career or aiming to climb the career ladder, JetBrains Academy has the programming courses to support you every step of the way.

www.jetbrains.com

It's a highly interactive, project based way of learning, that requires zero setup, so may appeal to those that want to jump straight in have their hand held the whole way - it's also free for the first 2 months, once you complete the first stage of a project then go to complete the first project. I think it's also interesting from a project perspective, as it appears to enable to get your teeth into something more meaty than the quick code challenges that I mentioned yesterday.

Will work through a few projects and post my findings on here.

 

[BigChazza]

Hi Sharky. Interesting thread. I'm on a similar path but a ways behind you. I have started looking at Quantopian an python. Not too intimidated by that because I have been a dev. But it's the linking it all together bit that concerns be - all the different packages and components you namecheck above. I need it in a big diagram!
Anyway, question: have you come across many examples of folks using another dev took for automated algo trading? E.g. C++, C# or Java? Thre are also some drag and drop data manipulation tools I know which seem capable of the repeatable task of pulling candlestick data from an API, processing it for signals and dealing with the results. But I guess you'd have to do all your logic by hand whereas with the python world you have all these libraries and external services for all the signal stuff? I guess my basic question is: do I really need to learn python, or do I just need to program (and be able to architect a cloud solution)?