Partners in Automation Software.

Then it went ok, even if the computer got turned off. But maybe it didn't get turned off, because you might be using a laptop or a UPS. What kind of UPS do you have? Or how do you deal with power outages?

These days I trade from home, and when the power goes off, I go out for a beer :LOL:

I do have a couple of UPS on the PC's, and in an emergency, access to a generator at the office. The longer I do this the simpler I try to make things. I suspect that the time is not too far away when I could probably do what I do by phone
 
I think that these discussions about various hardware solutions, hosting options etc should be based upon considerations of risk.

There is a cost based on mitigation of risk and a cost based on not miigating risk. These have to be weighed up in each personal circumstance.

The questions to consider are:
- what level of loss is acceptable ?
- how can the level of loss be limited to this amount ? e.g. initial order size, broker-side stops, fail-safe systems (UPS etc), other communication methods
- what is the cost of these systems in relation to the potential loss (monetary and other resources such as time)

What seems significant to me though is that the majority of the discussion within this thread has been focussed on hardware issues and virtually no discussion has taken place around issues which are more significant, in my view, with regard to profitability and risk of an automated system i.e. strategy and its physcial manifestation in the form of coding.

I think there is far more potential risk and loss associated with poor strategy and coding than there is with the odd hiccup from a power failure or hardware failure.

Charlton
 
Yeah, I agree. This big discussion on hardware and software all started when someone said "this can't be done from home": if you do things right, you can do it from home, all by yourself, too. That's my point. I see more problems coming from relying on others than advantages. The more people, the bigger the infrastructure, the more things can go wrong. Maybe two people is ideal.

Regarding this, I can tell you that I am working with another person, who worries about the risk and money management, while I worry about the creation and automation of strategies. That's right. I can't worry about everything - I admit it. I am incapable of double-checking my work for errors and risks, unless someone really insists. And then the errors appear. Right, because I am affected by wishful thinking when it comes to the work I do. I feel it's perfect as it is, and doesn't need to be fixed. When this person tells me "we need a way to monitor when some systems will fail", I feel like saying "how dare you? my systems don't fail...".

That's why I always lost money: wishful thinking. I was in denial about what didn't work, because my assumption was that everything was just perfect. Then when things went wrong, I got mad at the market (not at my systems), and engaged in revenge trading, and blew out my accounts, over and over again. This went on all the way to 2010, even after I was confident to have profitable systems, even after knowing I had profitable systems for two years. Even then, I still kept losing, due to bad risk and money management.
 
Last edited:
I know this is an old thread. Just wandering how the participants have progressed since then.

Travis you mentioned that you would be willing to help. Can you help by programming the API with IB?

Thanks
 
Well, each time I get asked this question, I always say in advance that I can give limited help, so wherever I said it, I must have added that part, too. It's like when I was a web designer and everyone asked me to build websites for them. I've done four or five, but then I had to cut down on it. The same applies to programming. My eyes are complaining. Eyesight is a limited resource. But of course, if by "help" we also mean advice and similar, feel free to ask me here or in private messages (better here, so everyone can learn and teach and ask). If instead by "help" we mean that i will code your systems on excel, then I can't do it. But I can answer specific questions, especially those on how to automate trading systems via excel, with IB as a broker.
 
Well, each time I get asked this question, I always say in advance that I can give limited help, so wherever I said it, I must have added that part, too. It's like when I was a web designer and everyone asked me to build websites for them. I've done four or five, but then I had to cut down on it. The same applies to programming. My eyes are complaining. Eyesight is a limited resource. But of course, if by "help" we also mean advice and similar, feel free to ask me here or in private messages (better here, so everyone can learn and teach and ask). If instead by "help" we mean that i will code your systems on excel, then I can't do it. But I can answer specific questions, especially those on how to automate trading systems via excel, with IB as a broker.



Thank you and I understand. Sadly my programming skills are limited so I would be unable to do any programming myself. It has been quite difficult to find a reliable programmer who just wants to do the job for what it is. Most are just ready to program your systems.

I am looking for someone who can create the link with the brokers for sending orders etc. I do have some strategies in excel but would not be able to automate these systems with IB as my knowledge is very limited.
 
Travis, a question that you may be able to answer as you are familiar with Interactive Brokers API. I read on their guide that the historical data request only goes back one year.

Is it possible to merge data beyond one year to this database?
Do you use their data for backtesting and demo trading?
In terms of account management, PL, account balance/margin etc ...do they allow you to collect this information via API already calculated or do the calculations have to be done in our own development?
Can we create custom timeframes of are we limited to only the ones they have?


Any advice would be appreciated.


Thanks
 
I will answer both posts, in the chronological order of your questions.

Sending orders is the easiest part. Their excel sample sheet is already set up for it, and you can download everything here:
http://www.interactivebrokers.com/en/p.php?f=programInterface

Installing the downloaded file wil create directories in your computer, and you'll find the sample (if you use windows) in this directory, more or less: "C:\Jts\Excel\TwsDde.xls".

I can help with that specific problem, and with other specific questions, but I can't code the whole thing for you: even with what i already know, it would take dozens of hours, let alone if one day you'll need changes, which you will for sure. So, before you get started you have to know if you have the time and are willing to learn excel enough to do most of it by yourself. I don't have a scientific background, and it took me about two years to teach myself the things I needed to code my systems on excel, studying several hours every day, and asking questions on various forums and to various friends (among whom the most helpful one is the guy who's running my server right now, at his house). If you're not going to do this incredible amount of work, and you still get started, then you'll end up relying on others for the next few years, and I can't do it, and each time you ask someone, you'll risk giving away your ideas. It's not a good idea to get started unless you're going to have this process entirely in your hands (except for those limited questions I was talking about).

Forget about the historical data from IB, because:

1) the data is only available for a year or so, which is not long enough for decent testing, unless you're creating systems that place several trades per day, in which case you'll still have a sample with hundreds of trades.

2) they don't adjust prices from contract to contract, so you'll end up with big steps from one contract to the next, and that is a problem I would not know how to solve. On contracts expiring once every 3 months it is already a big problem, let alone for contracts like CL and NG, which expire once a month. So, unless you're building intraday systems that do not trade overnight and that do not rely on yesterday's close, then you should discard this choice of using IB's data.

3) there is a reliable data vendor, DiskTrading Historical Data Download Server: FOREX, Indices, Futures - Intraday and Daily Market Data - Customer Download Accounts, which sells at very cheap prices (under 20 $ per symbol, and under 200 $ for all future symbols) all the data you need, and I've already verified that its data totally matches IB's data. I use that vendor, so this also answers your next question.

Now this penultimate question: "In terms of account management, PL, account balance/margin etc ...do they allow you to collect this information via API already calculated or do the calculations have to be done in our own development?". Yes, absolutely: you can look at the sample sheet I mentioned (go ahead and install it, as you don't even need an IB account to install it) and you will find sheets for that purpose (one sheet for monitoring the account balances and the margin, one for the trades, etc.).

Now, the last question: yes you can customize anything you want, because excel is very flexible. If you can do it manually, you can code it on excel. Then of course it's up to you to learn to do it, as I said before, even by asking questions step by step. Let's say that I can answer, a few at a time, another 50 questions like these five or so that you asked. Then I'd probably start getting tired, but obviously you can also ask other people. The most useful forum for this type of questions is, as far as I know, this one:
Forums - Automated Trading

The final objective, which I have accomplished, is to create something that you just need to turn on and off once a day. You could even manage to turn it on and off once a week, but there's more disadvantages than advantages, such as the risk of things going wrong, for an entire week at a time.

The quantity of errors I get, after several years of running my automated systems, is now about once a month, and it's always the same type of error. I turn on the systems at 16 US CT and once a month I find that the GBL, which opens 9 hours later, is not showing on TWS. So I have to restart TWS and that way it shows. That's why I do not recommend running the systems unattended for a week, because you might have this type of problem, and if you don't detect it, it will keep going for a week. And if there's an open trade, it will stay open for a week.

Another problem I forgot to mention, which I also see once a month or maybe a bit more, like twice a month, is the account balance. Sometimes it stops updating. But for my systems this is not a problem. And also, if you don't have the margin the trade will not be placed anyway, so monitoring the account balance is not very useful.

And these are not really my problems, but TWS problems.

I have finally solved other problems, bigger ones, and on my side, such as:
1) bugs in my excel programming (all solved in the first year of automated trading)
2) power outages (finally solved just 6 months ago)
3) connection failures (finally solved just 6 months ago)

I solved these last two problems by using a laptop and getting a secondary connection. It's all explained here:
http://www.trade2win.com/boards/trading-journals/140032-my-journal-3-a-11.html#post1749652

Then, after solving all the hardware and software problems, and after creating the automated systems (including years of back-testing on tradestation which we haven't even talked about), I am still having a lot of problems with my portfolio theory and risk management, but this is an entirely different field, which you weren't even asking about. But basically what I am saying is that this is a feat which will require several years. And that's why I can't do it for you, and you shouldn't get started unless you're going to work for years on it.

Here's another post I had written about the work required:
http://www.trade2win.com/boards/tra...elling-strategies-faq-help-2.html#post1701804
 
Last edited:
Is this not just a classic example of market forces at play. Travis needs capital to trade his systems. Others need programming expertise, Sounds like a match made in heaven.:)
 
Yeah, but I don't think that he wants to pay, so I didn't even ask him. Also, if he tells me his trading ideas, I can't promise that I will not use them myself. And finally, I don't like to get paid unless it's a large sum of money. I mean, I don't feel like giving him my details (name, last name) in order to receive a few hundred dollars on paypal. It's either a few thousands, or I'd rather do less for free. But once again, it would be better for him if he had everything in his own hands, by studying excel. You start with IB's sample sheet, "twsdde.xls", and then you modify it, little by little, and learn along the way.
 
Last edited:
Thank you for the advice and information. I will try to play around with the excel download.

In response to your other post, I don't mind paying. I have spoken to several programmers in the last few months. I have realized that this is a specialist field and someone with good experience or knowledge is required. They are all too interested in the systems and not really about the program. I have previously given the strategies to programmers at the beginning and all of a sudden they become traders. The last one spent weeks doing data mining etc to sees if he could improve it. For that period he wasn't focused on the programming.

It is not the excel side that I need programming . I have a bespoke platform that I am developing and the account management etc is required to be able to use that. The strategy builder and database structure is already there, all that is needed is for the data to start coming in, stored as time-frames, a few basic functions and it should be ready to go.
 
You're welcome. If it's not excel programming that you needed ("It is not the excel side that I need programming"), then I can't do it anyway, because I am only capable of doing it that way. I can't even call myself a "programmer" because I have a humanities background and I have taught myself "just" enough programming to be able to automate my systems on excel. If you have other questions related to automation on excel, let me know.
 
Last edited:
Thank you Travis. I appreciate all your input. You have already been a great help. If i have any more questions about the excel I will post them here.
 
Good and interesting discussion.

In terms of running a system once developed and tested, may I suggest a portable PC ( build-in UPS ) or a hosted server only you have control of. Using a portable at home requires a reliable internet connection. In case of a power faliure ( rare ) in your house, your PC still runs once power is back - unless we talk of hours of power cut. One could have a USB 3G modem atached the portable PC, to have the ping running, and to switch internet connection in case of power faliure ( Windows will switch automatically ).

A hosted server is about 1000 euro yearly. Both should have a ping watch dog to a web page. If the web page does not receive a ping after lets say 4 minutes, it will send you an SMS / email. Or when calculated and reported positions size dosnt match, or at no data feed. Ideally that you can log in to your system via the web page and see the status, and start/stop the system. An iPad with remote desktop is handy for accessing the full system.

In terms of developing a system for developing, testing and running a strategy - this should be the option when no other options available, and in full awareness that this takes years of hard work.

Having all the above, entrance barrier to automated trading is still the development and testing of the trading strategy. No idea, no strategy.
 
Last edited:
Hi everyone, I'm new here so, you can take this advice for what it's worth... For the past few years I've been an application architect for a fairly large company and just recently left to start building and running an automated trading system. My first suggestion is if you're really serious about building a rock solid system, that can run multiple trading strategies at the same time, have high availability(HA), disaster recovery(DR), extremely high volume and velocity and scale up and out... get a good architect involved to act as your technical lead and let him code and work with another programmer to make this happen for you. Not every programmer can act as an architect, it took me a while to learn to see the big picture!

Remember, "Performance is not an Add-on... it's designed into a system" so if you depend upon performance remember all of the built in latency you have in your internet connection probably doesn't make your home office a good place to run your server farm... at least not in the long run.

With that said, all of those decisions about the physical deployment should be made once you have a high level design or at least a choice made about the solution pattern that you're going to apply to the problem.

I hope this helps!

Sincerely,

Gary
 
Yes, thanks for the advice. From my point of view (I hope others will reply with theirs), things can also be done by just one person, with very limited financial means, like in my case. Actually maybe even better (cfr. E.P.Chang's book, chapter 8), due to the fact that... "If you want a thing done well, do it yourself".

Then if we're talking about high-frequency trading, things are as you described them. But not all automated trading is high-frequency trading.
 
Hi everyone, I'm new here so, you can take this advice for what it's worth... For the past few years I've been an application architect for a fairly large company and just recently left to start building and running an automated trading system. My first suggestion is if you're really serious about building a rock solid system, that can run multiple trading strategies at the same time, have high availability(HA), disaster recovery(DR), extremely high volume and velocity and scale up and out... get a good architect involved to act as your technical lead and let him code and work with another programmer to make this happen for you. Not every programmer can act as an architect, it took me a while to learn to see the big picture!

Remember, "Performance is not an Add-on... it's designed into a system" so if you depend upon performance remember all of the built in latency you have in your internet connection probably doesn't make your home office a good place to run your server farm... at least not in the long run.

With that said, all of those decisions about the physical deployment should be made once you have a high level design or at least a choice made about the solution pattern that you're going to apply to the problem.

I hope this helps!

Sincerely,

Gary

Hi Gary,

You wrote: and just recently left to start building and running an automated trading system

From a professional to a professional, I say forget it. Why re-invent the wheel ? With todays technology, most retail traders would be able to develop - in a simple and instant manner - sophisticated and complex strategies in minutes – which would take a team of programmers weeks/months. Every minute you use in trying to construct a system to develop and run trading strategies is 1 minute wasted - unless this is a sort of hobby, and you want to build something you know is your own, and be proud of you made, and dont mind spending some years or more on that - but aware that only You will ( perhaps ) be using it if finished one day. DOS migrated to Windows, ASM migrated to C and then to Visual Studio - programming of strategies and systems has migrated to visual development without programming. I even dare say to use one minute on programming a strategy is 1 minute wasted.

If however you decide to do it any way - you will face an almost endless walk needing to implement yet another feature, and then yet another feature. Hope this dosnt sound too harsh, but this is the reality.

Travis is right, such system can only be made by one person, a larger company would probably get stuck in meetings.
 
Last edited:
Yes, "stuck in meetings", just like E.P.Chan said here (page 159):
Quantitative Trading: How to Build Your Own Algorithmic Trading Business - Ernest P. Chan - Google Books

Another reason that independent traders can often succeed
when large funds fail is the myriad constraints imposed by management
in an institutional setting. For example, as a trader in a
quantitative fund, you may be prohibited from trading a long-only
strategy, but long-only strategies are often easier to find, simpler,
more profitable, and if traded in small sizes, no more risky than
market-neutral strategies. Or you may be prohibited from trading
futures. You may be required to be not only market neutral but
also sector neutral. You may be asked to find a momentum strategy
when you know that a mean-reverting strategy would work.
And on and on. Many of these constraints are imposed for risk
management reasons, but many others may be just whims, quirks,
and prejudices of the management. As every student of mathematical
optimization knows, any constraint imposed on an optimization
problem decreases the optimal objective value. Similarly,
every institutional constraint imposed on a trading strategy tends to
decrease its returns, if not its Sharpe ratio as well. Finally, some
senior managers who oversee frontline portfolio managers of quantitative
funds are actually not well versed in quantitative techniques,
and they tend to make decisions based on anything but quantitative
theories.
When your strategy shows initial profits, these managers may
impose enormous pressure for you to scale up quickly, and when
your strategy starts to lose, they may force you to liquidate the portfolios
and abandon the strategy immediately. None of these interferences
in the quantitative investment process is mathematically
optimal.
Besides, such managers often have a mercurial temper, which
seldom mixes well with quantitative investment management. When
loss of money occurs, rationality is often the first victim.
As an independent trader, you are free from such constraints
and interferences, and as long as you are emotionally capable to
adhere to the discipline of quantitative trading, your trading environment
may actually be closer to the optimal than that of a large
fund.
 
Top