Best Thread Algorithmic trading experiences

[north5]

I would have to agree with Pedro on this. I started with my own bespoke code - mainly because I wanted to use P&F analysis on forex and the standard packages didn't really offer it. However it became a real pain to implement my ideas this way, so that code has taken a back seat for now and I am working purely in MT4 at the moment. I have longer term plans to build that initial work into MT4 - but that is taking a back seat at the moment.

Building your own indicators is fine - but I'm dubious of the value. From my experience the actual indicator itself is really not that important. The importance lies in understanding what each indicator does and being able to build a program that is able to use them properly. For example does it matter if you use ATR or bollinger to measure volatility - my opinion is not really - what is more important is using the concept of volatility correctly.

The problem with an algorithm is also not all about in getting it to do what you want it to do - it's as much about it being able to have decide when it shouldn't do it. Normally this is called 'discretion' - but actually a person applies discretion when there is some facet in the market (be it volatility, excessive momentum etc) that an algorithm can also take into account. Provided, of course, that the person coding it understands the problem fully in the first instance.

 

[adamscj]

I see

At some point, you will come to the realisation that a lot of the stuff that is out there is of value.

For instance, on the TT forum, there is a fantastic piece of code for position sizing based on volatility. The theory (and a good theory at that) is that you should base the size of your position on the volatility of the instrument you are trading. If the instrument moves an average of $1 a day, you can stake more than if the instrument moves $10 a day.

This uses Average True Range, something that the trading world uses a lot.

With any indicator, there's a thousand ways to use it - so don't think that using an indicator as part of your strategy means that someone else will have a similar strategy to you.

It seems you are saying that not ony will you code everything from scratch but that you will not benefit from the research that thousands have done before you. This leaves you with an even larger task.

I would presume that volatility based position sizing is something you hadn't considered yet and I think it's safe to presume it's something you may never have come up with on your own, yet it is a very smart idea. There are hundreds of such things you need to know about and incorporate into your strategies,depending on how you trade.

I fully understand the 'go it alone' mentality but you will eventually come around to using mainstream tools & methods as they exist for a very good reason.

Actually, I've already incorporated position sizing based on volatility into my model - you can see my previous posts where I've recommended it as an approach to others.

I certainly agree that understanding & using other research out there is vital to any success I might have. I'm not niave enough to believe that I can come up with a winning strategy that no-one else has ever thought of. I have however worked in the investment industry for some time and therefore use my experience to fine tune & improve the models I've encountered along the way. To do this, I'd rather write something myself than try to use a piece of software that doesn't quite do what I want or is difficult to enhance

 

[pedro01]

The fact that TS doesn't quite work in the way I want is a minor inconvenience compared to the amount of stuff it does give me. For the stuff it doesn't give me, I can write it myself & call it from TS.

It's the old 80/20 rule - 20% of the work will give you 80% of the benefit. There's simply no benefit in developing this from scratch. I don't even have 20% of the work as the software gives me most of what I want. In my case, all I need to code are rules and some nifty lines of code to share data between the 50 or so charts I monitor in real time.

Perhaps you could give us some tangible benefit in doing what you are doing or a list of things you are doing that TS, MT4, Ninja Trader, e Signal etc don't give you out of the box.

Personally (and this is with 20 years in the Software industry), I just don't see it.

Each to his own though.

 

[AnEdge]

If you are serious about developing a real quality automated system then check out the open source java platform at Marketcetera Trading Platform: index .... but you will need to be a software professional to use it as it is aimed at small hedge funds which would have a FT developer to code up the ideas.

 

[dcraig1]

If you are serious about developing a real quality automated system then check out the open source java platform at Marketcetera Trading Platform: index .... but you will need to be a software professional to use it as it is aimed at small hedge funds which would have a FT developer to code up the ideas.

Have you used it at all ? If so any comments on it's strengths and weaknesses ?

Java has got to be a real plus - open source and platform independent.

 

[AnEdge]

I am testing it at the moment...it is early stages, so it is a little rough around the edges but the core software is really solid. I have downloaded all source code and have been going through it.

The only thing that comes close is Welcome to ActiveQuant! | ACTIVEQUANT .

If you want a real OMS with FIX connectivity and the ability to run multiple tick-by-tick strategies...marketcetera seems to be the way to go, it will provide you the underlying platform for you to build upon. BUT you are working with an early life-cycle open source product...eg. no bells and whistles

 

[pedro01]

Hmmm - I am dubious about this - it's designed for small hedge funds yet is in its infant stages ? Isn't "designed for small hedge funds" just a way to try to pull traders in ? Sounds like a marketing statement to me.

I checked the web sites, it says there's documentation for it but I can't see any when I click the links.

AnEdge - are you affiliated with the producers of this software in any way ? If no - where did you go to get docs on the features.

I pretty much finished the main part of my TS strategy today and would be happier with something with better order management facilities. I'm not seeing any info on these products that would make me want to try them yet. Don't they have an overview ?

As for Java = good. Java is fast turning into a dying language in my point of view. I think .NET will kill it off over the coming years for mainstream development as Java simply leaves you with too much work to do outside of the actual coding... at least with .NET you get a full application framework, not have to bolt it together from lots of different sources...

Flame on....

 

[robster970]

As for Java = good. Java is fast turning into a dying language in my point of view. I think .NET will kill it off over the coming years for mainstream development as Java simply leaves you with too much work to do outside of the actual coding... at least with .NET you get a full application framework, not have to bolt it together from lots of different sources...

Flame on....

LOL. I have 20+ years of engineering/software development too. I also ran my own software company and sold in 2004. I might not know much about trading but I certainly know a lot about software engineering and what to use to build apps pertinent to the environment they'll be used in.

Java dying? That's like saying C++ is dying. Like any engineer you use the best tools for the job, not just pick the tool and then decide what to build afterwards.

There is no way on earth that anybody building a high performance server based system for use in a telecommunications/aerospace/finance environment that needs real grunt would use MS framework to do it. I can tell you this from first hand experience of build commercial server apps in both .NET framework and using Java in the various App server containers that are available both free and paid for.

Consider yourself flamed

Agree with rest of what's said though. Wouldn't trust Open Source app like that because the developer community is not big enough to sustain it.

 

[dcraig1]

As for Java = good. Java is fast turning into a dying language in my point of view. I think .NET will kill it off over the coming years for mainstream development as Java simply leaves you with too much work to do outside of the actual coding... at least with .NET you get a full application framework, not have to bolt it together from lots of different sources...

Flame on....

OK then, I will. I think thats a very silly statement.

.NET runs on Windows. And no amount of Mono is ever going to truely make it in the OSS / *nix environment. As a lot of server infrastructure runs *nix and most assuredly will continue to run *nix, it follows that .NET is not going to displace Java in that context. .NET is an extremely poor second to Java in the mobile device area. Can't see that changing anytime soon either.

It is a virtue having many different sources for software components. When that dies, innovation dries up. The world has leared something from open source.

 

[dcraig1]

Lets take an example of a trading system/charting product developed in Java targetted at a PC environment. What does Java SE lack that you get in .NET ? I have develped a lot of trading stuff in Java including most of the elements of a commercial product, and I'll tell you what external libs I have needed:

1. Charting - I use JFreeChart with some real time performance enhancements of my own. Does .NET give you this - bet it doesn't. No advantage for .NET there.

2. Database/filestore. I use MySQL - free fast and reliable. Of course there may well be better solutions for huge amounts of tick data, but you wont find them off the shelf in .NET.

3. XML handling. I could use the inbuilt stuff, but use JDOM as it's just plain easier.

4. Genetic Algorithms. JGAP. Off the shelf in .NET ?

5. IB and opentick APIs for data and/or execution.

6. JTidy for cleaning HTML pages and converting to XML.

7. Some free icons.

8. A free command line parser enforcing strict *nix command line arg rules.

And last but not least Netbeans IDE.

Please explain how .NET would have made a tap of difference in using third party libs, or in fact would have to been superior to Java in any way except possibly easier GUI development than Swing.

Total cost of software - $0. And it runs on a superior operating system.

 

[robster970]

2. Database/filestore. I use MySQL - free fast and reliable. Of course there may well be better solutions for huge amounts of tick data, but you wont find them off the shelf in .NET.

It's even faster if you run it in memory and thus stopping IO. Think of it as caching. Memory is cheap these days. Consequence of doing this is SAR on the machine drops therefore making processing even quicker too. You should try it and see what happens.

 

[dcraig1]

A quote from a very unsatisfied user of off the shelf products over on ET:

"I have pretty much driven myself close to insanity with these various automated backtesting and trading systems. Everything from NinjaTrader, to RightEdge, to whatever. None of them have built in debugging. Does that strike you as basically insane?

I am having to recompile, put print statements everywhere, and its just zapping all the energy I have for this.

Does anyone know of a platform that has built in debugging and doesn't cost $80k a month?

Sorry I am freaking out here I have just had it with these half baked platforms."

I can relate to that. When I first looked at eSignal, I had the same reaction about no debugger, and complete digust at no library facility (now fixed apparently - but how long does it take ?).

I'm not advocating roll your own over off the shelf products - it depends on your aims, your capabilities and many other things but it is an unwarranted assumption that vendors necessarily know what they are doing at a really expert level.

 

[pedro01]

Lets take an example of a trading system/charting product developed in Java targetted at a PC environment. What does Java SE lack that you get in .NET ? I have develped a lot of trading stuff in Java including most of the elements of a commercial product, and I'll tell you what external libs I have needed:

1. Charting - I use JFreeChart with some real time performance enhancements of my own. Does .NET give you this - bet it doesn't. No advantage for .NET there.

2. Database/filestore. I use MySQL - free fast and reliable. Of course there may well be better solutions for huge amounts of tick data, but you wont find them off the shelf in .NET.

3. XML handling. I could use the inbuilt stuff, but use JDOM as it's just plain easier.

4. Genetic Algorithms. JGAP. Off the shelf in .NET ?

5. IB and opentick APIs for data and/or execution.

6. JTidy for cleaning HTML pages and converting to XML.

7. Some free icons.

8. A free command line parser enforcing strict *nix command line arg rules.

And last but not least Netbeans IDE.

Please explain how .NET would have made a tap of difference in using third party libs, or in fact would have to been superior to Java in any way except possibly easier GUI development than Swing.

Total cost of software - $0. And it runs on a superior operating system.

Pretty much proves the point - look at all the bits you had to putt together to get your system together.

The problem that Java will face in the large application world is all the choice. It's a nerds tool more than a true business application development framework.

The future in my opinion is Microsoft/.Net/Azure & SaaS.

Time will tell. I just think that building large scale commercial apps on Java is going to die out. Of course there will be die hard fans - but let's see where Java is in 10 years time. My opinion is it's going in a million directions but getting nowhere.

You can't design by committee & that's what will ultimately be the death of Java.

 

[pedro01]

A quote from a very unsatisfied user of off the shelf products over on ET:

"I have pretty much driven myself close to insanity with these various automated backtesting and trading systems. Everything from NinjaTrader, to RightEdge, to whatever. None of them have built in debugging. Does that strike you as basically insane?

I am having to recompile, put print statements everywhere, and its just zapping all the energy I have for this.

Does anyone know of a platform that has built in debugging and doesn't cost $80k a month?

Sorry I am freaking out here I have just had it with these half baked platforms."

I can relate to that. When I first looked at eSignal, I had the same reaction about no debugger, and complete digust at no library facility (now fixed apparently - but how long does it take ?).

I'm not advocating roll your own over off the shelf products - it depends on your aims, your capabilities and many other things but it is an unwarranted assumption that vendors necessarily know what they are doing at a really expert level.

Tradestation has a built in debugger - all you have to do is put in breakpoints

 

[pedro01]

It's even faster if you run it in memory and thus stopping IO. Think of it as caching. Memory is cheap these days. Consequence of doing this is SAR on the machine drops therefore making processing even quicker too. You should try it and see what happens.

Good point - all trading software should be memory resident - just like mainstream planning software used in factories every day.

You can't do that much number crunching any other way.

 

[blackcab]

Nice geeky thread For what it's worth I'm another one doing my own home-brew thing. I'm writing a backtest program on one hand, and have been studying some machine learning methods for quite a while on the other, which I'll use to generate or evolve systems to test. One reason I like writing my own code is that it's completely open-ended, if you want to test a mad idea combining moon phases and weather and non-linear transformations, you can. My main problem is time, ie not having any. Writing this post has used half my free time for today! When the buggers leave home I'll have my life back, rant, rant, etc...

 

[pedro01]

If any of you guys want a peek at Tradesation one day - I would be willing to show you it & walk you through what it does.

I have access to webex for screen demo's & we can use Skype for voice.

I will not be trying to sell you anything - just showing you what it does & answering any questions you may have.

Let me know if you want a peek.

 

[zupcon]

One reason I like writing my own code is that it's completely open-ended, if you want to test a mad idea combining moon phases and weather and non-linear transformations, you can.

After wasting years developing with wealthlab, tradestation, metatrader, and a bunch of other stuff I have to agree writing your own solution (for system design purposes at least) is infinately more sensible in every concievable way (other than time and/or money required !)

Commercial products are great for checking out simple ideas quickly, unfortunately their development is driven by the requirements of their users, and most of the users of these tools dont really know what they want, which pretty much results in tools that dont really do very much.

If you need database support, or AI, or specialist statistical analysis techniques, its simple enough in most development environments, try doing it with a commercial trading application and you invariably end up with a clunky fudged solution.

If I where starting from scratch, I'd probably go down the open source Java route (rather than .NET which was the easiest and quickest solution based on experience), anything that involves money is not to be trusted to a microsoft operating system (and unfortunately I have the scars to prove it )

 

[robster970]

So. My little ATS has been busily pretending to buy and sell this week using FTSE. It's working out profit factor, average win, average loss, win ratio.

Over what number of trades do people tend to run new strategies to get statistically meaningful results?

 

[pedro01]

It seems we have 2 camps

1 - use existing tools as they do most of the work for you
2 - start from scratch

Now - there is a middle way. You can link DLLs to Tradestation, so if you have a burning desire for a feature that isn't there, you do not need to re-write the whole product, just that feature.

As I have been posting here I have also been working on my system. I use a DLL to handle global variables to pass data between the 2 x 40 radar screens & 6 charts that make up the strategy which is pretty complex.

I feel that perhaps people are expecting these tools to be something they aren't.

These tools will not comeup with a strategy for you. They will not sit there analyzing the markets looking for patterns to find the Holy Grail.

If you have a strategy or want to try out strategies, these tools are fantastic with the proviso that you need to be a bit of a coding whizz as well as use some lateral thinking to fit your requirements around the timing cycle of the software. This is no different to getting to grips with any language in my opinion.

As for artificial intelligence - that is certainly beyond the scope of these tools and beyond the scope of what you need in order to make money.

I would like to know from the 'ground up' people specifically what types of things they were trying to do when they gave up - let's not be vague here - get specific and let's discuss what it is you think these tools should do that they can't.

From where I sit, I've managed to do everything I wanted so far without having to write a trading API, tick data collection module, visual representation tools, strategy reports. graphs, charts, demo account management, order tracking etc. - I get all that out of the box.

I also get access to thousands of other people using the same toolset and access to lots of open source ELS code that helps me in my endeavours.

So - what is it exactly that you can't do that makes you want to write all that other stuff ?