Adamus
Experienced member
- Messages
- 1,898
- Likes
- 97
The brief extract from "Algorithm Design" that I read on Amazon looks quite intense reading for someone who hasn't even learnt a programming language yet!
In fact it makes me realise that despite my career to date as a programmer, I'm not interested enough in software development to devote time to reading it because I devote all my time to trading relating business.
I wouldn't recommend that book because I think it's too far up the learning curve unless you're a professional software developer. I'm not saying I know how to teach algorithm writing skills, in fact I think it's probably incredibly difficult. It's basically an advanced form of problem solving.
On the other hand maybe the data structures part of 'Algorithm Design' is useful, the extract didn't go into it at all. I'm currently wrestling with EJB JPA Hibernate, and I wish I could optimise it, but I think it's an inherent limitation of the object - relational mapping paradigm that's giving me grief.
You do this stuff for a living, don't you? I think you mentioned it vaguely at some point. I'm saving trades, profits and running totals from different analyses with various commission rates, points spread, slippage, ATR etc, and so there are about 20,000 records to be written to the database for each analysis but even so it takes about 3 or 4 minutes which seems unreasonable for dual 1.5GHz CPUs.
In fact it makes me realise that despite my career to date as a programmer, I'm not interested enough in software development to devote time to reading it because I devote all my time to trading relating business.
I wouldn't recommend that book because I think it's too far up the learning curve unless you're a professional software developer. I'm not saying I know how to teach algorithm writing skills, in fact I think it's probably incredibly difficult. It's basically an advanced form of problem solving.
On the other hand maybe the data structures part of 'Algorithm Design' is useful, the extract didn't go into it at all. I'm currently wrestling with EJB JPA Hibernate, and I wish I could optimise it, but I think it's an inherent limitation of the object - relational mapping paradigm that's giving me grief.
You do this stuff for a living, don't you? I think you mentioned it vaguely at some point. I'm saving trades, profits and running totals from different analyses with various commission rates, points spread, slippage, ATR etc, and so there are about 20,000 records to be written to the database for each analysis but even so it takes about 3 or 4 minutes which seems unreasonable for dual 1.5GHz CPUs.
Quick update....
Have spent most of December in vary states of illness. Got some sort of demon chest infection, and while the bulk of it cleared quickly enough it wasn't until a couple of days ago that it finally completely went. Now I'm less ill, I'm also very much aware of how much I should have done something about the EUR/GBP rate soaring over the last few weeks. My savings are now worth quite a lot less, and I'm not impressed.
GBP should rebound any minute now, right?
Back to the topic at hand; haven't managed to get down to 3/5ths time as planned, so am not going to have free time for this anytime soon. Will update as I know more, but right now I'm mostly focused on damage control for the financial disaster I think is coming.
Have spent most of December in vary states of illness. Got some sort of demon chest infection, and while the bulk of it cleared quickly enough it wasn't until a couple of days ago that it finally completely went. Now I'm less ill, I'm also very much aware of how much I should have done something about the EUR/GBP rate soaring over the last few weeks. My savings are now worth quite a lot less, and I'm not impressed.
GBP should rebound any minute now, right?
Back to the topic at hand; haven't managed to get down to 3/5ths time as planned, so am not going to have free time for this anytime soon. Will update as I know more, but right now I'm mostly focused on damage control for the financial disaster I think is coming.
Ooops, sorry forgot to reply to this earlier!
Yes, I'm a full time computer science researcher, primarily working as a developer, but I also do sys-admin work and obviously bits of paper writing, presentations etc. here and there.
I've seen systems write 20k records back to a database in under a second, it shouldn't be a serious issue. My immediate thoughts are:
1. Maybe Hibernate really screws things up.
2. You're not reconnecting every time or something are you?
3. Prepared statements might help, where the server pre-parses the INSERT command and you just queue up new data to be written.
Not sure what technical level to write this at, did that help?
Adamus
Experienced member
- Messages
- 1,898
- Likes
- 97
Thanks for the help - technical level probably a bit below what I can deal with - Hibernate is an object-relational mapping tool, so it should have taken care of (2) and (3) itself. That leaves (1) which is probably correct!
Sorry about your illness! Bad luck. Try echinacea - i think it's meant to boost your immune system - and also could it be a house dust / dust mite allergy? Just discovered I have that allergy myself.
Re EUR:GBP - not it's probably not going to bounce back - it'll come back one day, but not as quick as it got there. IMHO. Just look at the Bundesbank -- er I mean the ECB.
Sorry about your illness! Bad luck. Try echinacea - i think it's meant to boost your immune system - and also could it be a house dust / dust mite allergy? Just discovered I have that allergy myself.
Re EUR:GBP - not it's probably not going to bounce back - it'll come back one day, but not as quick as it got there. IMHO. Just look at the Bundesbank -- er I mean the ECB.
Yeah, should try echinacea. I've been taking garlic capsules too, they're meant to help...
EUR/GBP does appear to have mysteriously bounced, although right now I'm just thinking of giving up on trading. The entire last year has gone badly wrong. At best I don't have the nerve to scale up my trading beyond dabbling with spread betting, at worst I was repeatedly lucky throughout 2007.
Adamus
Experienced member
- Messages
- 1,898
- Likes
- 97
I think you don't have the nerve because you don't have the test results to give you enough confidence. Sure you have test results, but your subconscious wants more.
From what you say, it sounds like you've developed a useful mechanical trading system, but your algorithm might not be up to scratch in terms of the robustness you need, perhaps smaller and shorter drawdowns, to stop you from pulling it out of the markets.
It took me ages to find an algorithm that was even remotely profitable, and since I did no work on programming an execution interface with IB's API, I do it all by hand. It sounds like you spent a large amount of time on the execution app, but need a better algorithm, something that is more consistent, even with more slippage built-in.
I'm back, and I'm all about the fundamentals
Two months break and I'm back! So, what am I doing differently...
Well, I've come to the conclusion that pure technical analysis probably isn't worth it for me. What you're essentially doing is looking for optimisations that a human trader will miss, and to be honest that's relatively small stuff right now, compared to what you can make doing fundamental analysis. Still getting back up to speed (LOTS of news articles to read), but I've "un-lost" £150-ish over the last 3 days (which is fairly good going for a casual trader, I feel).
Long term, still very positive about the idea of trade-by-wire; my biggest losses/missed profits by far are from typing/thinking errors, or by being too busy with the day job. If I can have a computer program take general direction from me and then execute the details itself, I think I can do much better.
Expect a more detailed post over the weekend.
Two months break and I'm back! So, what am I doing differently...
Well, I've come to the conclusion that pure technical analysis probably isn't worth it for me. What you're essentially doing is looking for optimisations that a human trader will miss, and to be honest that's relatively small stuff right now, compared to what you can make doing fundamental analysis. Still getting back up to speed (LOTS of news articles to read), but I've "un-lost" £150-ish over the last 3 days (which is fairly good going for a casual trader, I feel).
Long term, still very positive about the idea of trade-by-wire; my biggest losses/missed profits by far are from typing/thinking errors, or by being too busy with the day job. If I can have a computer program take general direction from me and then execute the details itself, I think I can do much better.
Expect a more detailed post over the weekend.
Frustrated.
I keep making stupid mistakes. Today, overtrading. I aim to do 2-4 trades a day, and ended the day with, drumroll please... 24 roundturns. TWENTY FOUR. Thursday, I bought instead of sold and then hung onto the trade. Overtrading is a common one for me, though.
Tired, and frustrated, and need to take a break. Far too much logging into the trading system and going "Okay, what can I trade?" rather than "Okay, what's too expensive/cheap?". Given the screwups I'm making, I think I'm doing relatively well, but deeply, DEEPLY need to take more care before I bankrupt myself.
Intending to get MySQL support written into JBookTrader as soon as I have more time, and generally develop it closer to the codebase I want it to be.
I keep making stupid mistakes. Today, overtrading. I aim to do 2-4 trades a day, and ended the day with, drumroll please... 24 roundturns. TWENTY FOUR. Thursday, I bought instead of sold and then hung onto the trade. Overtrading is a common one for me, though.
Tired, and frustrated, and need to take a break. Far too much logging into the trading system and going "Okay, what can I trade?" rather than "Okay, what's too expensive/cheap?". Given the screwups I'm making, I think I'm doing relatively well, but deeply, DEEPLY need to take more care before I bankrupt myself.
Intending to get MySQL support written into JBookTrader as soon as I have more time, and generally develop it closer to the codebase I want it to be.
Adamus
Experienced member
- Messages
- 1,898
- Likes
- 97
I had a look at JBookTrader myself but isn't the whole thing restricted to trading the book of bids and asks? I know nothing about that and left their website, pronto. Am I mistaken? I take it you're still doing forex - how does that actually work with JBookTrader?
Interactive Brokers have a trading book feed for currency markets. Obviously, it's not complete, but it would be at least enough to make JBookTrader work.. In the end, have had other things keeping me busy, and never got around to it...
Thread necromancy powers ENGAGE!
Okay, but seriously... back trying to find time to code. Current intention is to go back to my codebase (JBookTrader is heading more in the drection of automated scalping, while my approach is closer to day trading). The entire trading engine needs re-working; I've been trading by hand using sets of three orders; one entry, two closing (limit & stop) orders triggered on the first, and the intention is to reproduce that.
Right now, the major task is to adapt the strategy interface so that it can establish entry prices, as well as exit, ahead of time. It needs to be able to exit quickly in case of high profit, or if the market swerves hard away from a profitable outcome. Also need code to handle order issues (orders failing to execute, accidental cancellation, etc.)
Bad news is, I'm going to be using this stuff as portfolio code, so it's not going to be freely available any more. Even if I can't ever make this work with my own resources, the idea of going into full time development work on trading platforms appeals, so I'm going to be looking for jobs in the area, and figure my own platform is a good start!
Okay, but seriously... back trying to find time to code. Current intention is to go back to my codebase (JBookTrader is heading more in the drection of automated scalping, while my approach is closer to day trading). The entire trading engine needs re-working; I've been trading by hand using sets of three orders; one entry, two closing (limit & stop) orders triggered on the first, and the intention is to reproduce that.
Right now, the major task is to adapt the strategy interface so that it can establish entry prices, as well as exit, ahead of time. It needs to be able to exit quickly in case of high profit, or if the market swerves hard away from a profitable outcome. Also need code to handle order issues (orders failing to execute, accidental cancellation, etc.)
Bad news is, I'm going to be using this stuff as portfolio code, so it's not going to be freely available any more. Even if I can't ever make this work with my own resources, the idea of going into full time development work on trading platforms appeals, so I'm going to be looking for jobs in the area, and figure my own platform is a good start!
Have re-activated my IQFeed subscription (BTW, if anyone is thinking of subscribing, I'd love to get the referral credit ), and paid for the dev kit ($300 down the drain, oww). Haven't had time to be coding, but have been trying to trade my system by hand where I can (and mostly getting kicked by the market moving faster than I can trade by hand).
Today, a reminder on the importance of stop-losses:
Bad news (for me) is I was long on GBP when the 3rd quarter GDP news hit (and it wasn't pretty). Good news is that while my stop-loss slipped 20 points before it managed to trade, it's still a hell of a lot better than the 100+ points it would have cost me without one.
), and paid for the dev kit ($300 down the drain, oww). Haven't had time to be coding, but have been trying to trade my system by hand where I can (and mostly getting kicked by the market moving faster than I can trade by hand).Today, a reminder on the importance of stop-losses:
Bad news (for me) is I was long on GBP when the 3rd quarter GDP news hit (and it wasn't pretty). Good news is that while my stop-loss slipped 20 points before it managed to trade, it's still a hell of a lot better than the 100+ points it would have cost me without one.
Dealing with bit-rot
Holy new release Batman!
Still dusting the mess off, updating libraries, and generally cleaning the mess up. I'll have a proper release (with instructions and everything) in the next few weeks, but for now here's an update for anyone who was dabbling with the really old versions:
http://lostics.org/foxquant-distrib/foxquant-20091024.zip
Key changes:
Holy new release Batman!
Still dusting the mess off, updating libraries, and generally cleaning the mess up. I'll have a proper release (with instructions and everything) in the next few weeks, but for now here's an update for anyone who was dabbling with the really old versions:
http://lostics.org/foxquant-distrib/foxquant-20091024.zip
Key changes:
- The project is now "Fox Quant", because it sounds better and makes it easier to explain if it ever moves away from currencies.
- Updated JCommon, JFreechart, iText, POI and MySQL Connector/J libraries
- Removed custom IB Trader Workstation API implementation, and replaced it with the latest official API
- Removed Collective2 - as implemented it was more of a liability than an asset.
Hammering away at the code, and slowly feeling like things are beginning to make sense. Had a lot of trouble figuring out how I wanted tick data stored (it comes in as a single price or size value at a time from Interactive Brokers; if I store that directly it's a nightmare to play back, but if I store full price/size set at each point there might be inaccurate records).
What I'm really noticing is that as I stop trying to hard complexity from the strategy, The first version had a single "Here's some new data" method that was called, and that then called the strategy runner to buy/sell. The new version has four methods:
backfillTick() is used during warmup to add data for the strategy, without being tradeable. Previously I passed an "is up to date" parameter in, which was more error prone as the strategy could still try trading if it didn't check that parameter.
The remaining three methods all execute trades by returning them. This IMHO is a lot simpler than the previous version where methods on another object were called, and makes the actual flow clearer. handleTickWhileFlat() returns an object that contains a limit order for entering the market, and a pair of orders (stop and limit) for exiting. The strategy runner then pushes them through to the market as a complete set, with the second two triggered on the first.
The handleTickWhileLong() and handleTickWhileShort() methods return updated values for the stop/limit orders.
The result is that the strategy has a lot less conditionals in it, with much of the logic ending up in the code structure instead. I think it's better anyway...
What I'm really noticing is that as I stop trying to hard complexity from the strategy, The first version had a single "Here's some new data" method that was called, and that then called the strategy runner to buy/sell. The new version has four methods:
- public void backfillTick(final long tickTime, final Double bidPrice, final Double askPrice, final Integer bidSize, final Integer askSize);
- public EntryOrder handleTickWhileFlat(final long tickTIme, final Double bidPrice, final Double askPrice, final Integer bidSize, final Integer askSize);
- public ExitOrders handleTickWhileLong(final long tickTIme, final Double bidPrice, final Double askPrice, final Integer bidSize, final Integer askSize);
- public ExitOrders handleTickWhileShort(final long tickTIme, final Double bidPrice, final Double askPrice, final Integer bidSize, final Integer askSize);
backfillTick() is used during warmup to add data for the strategy, without being tradeable. Previously I passed an "is up to date" parameter in, which was more error prone as the strategy could still try trading if it didn't check that parameter.
The remaining three methods all execute trades by returning them. This IMHO is a lot simpler than the previous version where methods on another object were called, and makes the actual flow clearer. handleTickWhileFlat() returns an object that contains a limit order for entering the market, and a pair of orders (stop and limit) for exiting. The strategy runner then pushes them through to the market as a complete set, with the second two triggered on the first.
The handleTickWhileLong() and handleTickWhileShort() methods return updated values for the stop/limit orders.
The result is that the strategy has a lot less conditionals in it, with much of the logic ending up in the code structure instead. I think it's better anyway...
NetTecture
Well-known member
- Messages
- 405
- Likes
- 12
Just as a note, I saw something similar and found out that NInja is pretty bad in some things.
* NInja does backtest bar by bar, not tick by tick. THis means executions are not necessarily too relevant. Part of the API even doesn ot work at all in backtest mode (those to intercept ticks executing or market updates).
* Even if going down, it is a tick by tick analysis at best. THere is no relevance to bid/ask. In some market conditions this means executions can vary widely from what backtest shows.
I got into serious problems backtesting / coding a strategy using trailing stops evaluated on a per tick bases - basically, backtest can not properly test that.
Some other framework (TIckZoom, one I work on) are backtesting on tick streams - even Tradestation does so (I.e. look into the bars when trades executes). My own fframewokr on top will not only use trades, but also bid/ask, to make sure market orders are PROPERLY priced (i.e. not next tick, but best bid/ask available).
So, depending on what the difference is.... it can be a technical issue with the NInjaTradert shortcomings.
The other thing is, NinjaTrader tries to hide all the complexity of what's going on, which... well, from what I've seen so far, mostly enables people to shoot themselves in the foot. My API used to be similar to the NinjaTrader one, but is now moving across to a model where it takes in not just a mid-point, but bid/ask prices and bid/ask size. This also handles more freaky situations such as where there is no bid/ask price!
This is primarily so I've got an off-site backup, but there's a snapshot of the latest code at http://lostics.org/foxquant-distrib/foxquant-20091028.zip
A lot of the interfaces (API) are simpler, which is nice. Tomorrow I get to work through the actual order execution code, make sure my understanding of how to get bracket orders into Interactive Broker's system is correct!
A lot of the interfaces (API) are simpler, which is nice. Tomorrow I get to work through the actual order execution code, make sure my understanding of how to get bracket orders into Interactive Broker's system is correct!
Basic trading engine is back in place. It's by no means done, but I've attached today's test trades for the curious. At the moment it can't handle trade status changes, so it won't ever re-enter the market after a trade finishes, so that's most of my work for the weekend. Then finishing database changes, getting backtesting to work again...
