Signalcalc
Veteren member
- Messages
- 4,670
- Likes
- 1,031
How are you avoiding the issues of curve fitting and data snooping in your testing?
Good question.
Take a look at the first post in the series of posts dealing with developing and validating the system from last Sunday. From this you will see that the two filters I have applied to the system in addition to the basic stock selection filter I am using (which should be pretty clear is simply seeking stocks trending up), both have plausible explanations for why they work.
These filters are generally the first thing I try after running a backtest to see if I should bother spending more time developing a system. So, I didn't really so much look at the data and decide that I should try these filters since it will make my results better, it was more, I tried them because they have a valid reason for working.
Certainly it could be argued that I curve fit the specifics of the filters (i.e. first 5 minutes, greater than $50) to some extent. Although really when I am trying to get the specifics of filters I try to be reasonable in the limits I test on them. What I mean by this is, I haven't gone through and tested stock greater than "x" where "x" equals every value from 1 to 100. I took much broader limits for this --> going by increments of 10. Each increment encompassing a fairly large percent of the unfiltered systems trades. For example the greater than $50 contains ~57% of the total unfiltered system trades and the greater than $60 contains ~48% of the total unfiltered system trades - so about 9% of the system trades lie between $50 and $60.
Same thing for the time filter. I decided on the time increments based on trying to keep the number of trades in each time interval fairly close. I didn't want to break the first two minutes down minute by minute which is why that grouping is larger than the others. Perhaps the best interval for the in sample testing was actually anything within the first 7 minutes. I don't really know or care, because taking it to that level would certainly be curve fitting to an extent. I would likely end up with similar results to the validation run however which is sort of the point.
Setting my floor at anywhere from $45 to $65 will likely yield approximately the same results. Likewise, setting my time stop anywhere in the first 4 to 7 or so minutes would likely yield similar results. This seems like a fairly restrictive interval, but really 70% of the total trades of the unfiltered system occurred within the first 10 minutes of the day, so there is a pretty large difference between a 4 and 7 minute time stop.
The way I checked to make sure I didn't curve fit/data snoop my results was by checking against data that was not touched when I developed the additional filters. The results from this validation run were comparable to the results from my initial run when developing the filters, so I feel comfortable saying that I have not overfit the development data, and the developed filters should hold into the future.
Although there is not as much data for the validation set due to fewer trade opportunities, I did check and see the "optimal" values for the two filters with that set as well. As is turns out, the validation set came up with an optimal time stop of enter the trade within the first three minutes (versus five) and trade only stocks greater than $50 (matches initial run finding). So very similar results between the in sample and out of sample.
Really the only way I can see this system falling flat on its face at this point is if I am unable to get my limit orders filled. I figured out a few tricks from my last live test run to get fills, so we will see how they play out for this one. I tried to address this in my backtesting by forcing the price to trade through my limit order before my limit order was considered executed (at the limit price), so we will see...
Simulation trades for this week meeting my price and entry filters:
06/12/2017 Trades
SPY - for context
AON
Entry/Exit not showing up for some reason, but they were:
Entry: $132.46
EOD Exit: $132.90
Net: +0.588%
Normalized Day Net: +58.8%
06/13/2017 Trades
SPY - for context
MAA
Entry: $132.46
TP Exit: $132.90
Net: +1.895%
Normalized Day Net: +189.5%
06/14/2017 Trades
SPY - for context
ODFL
Entry: $132.46
SL Exit: $132.90
Net: -0.969%
AON
Entry/Exit not showing up for some reason, but they were:
Entry: $133.77
TP Exit: $134.94
Net: +1.530%
Normalized Day Net: +28.1%
06/15/2017 Trades
SPY - for context
No trades taken.
Normalized Day Net: +0.000%
06/16/2017 Trades
SPY - for context
AON
Entry: $135.76
EOD Exit: $136.36
Net: +0.870%
Normalized Day Net: +87.0%
Overall Week Net (equal risk each day): 363.4% of daily risk - too bad I didn't start my live testing this week...a phenomenal week for my system...hopefully my actual entries will match up with my theoretical entries this time around.
The ability to edit posts disappears rather quickly...I forgot to update the entries and exits in some of the trades I posted above. Percentages are correct, but the entries/exits were as follows (corrected values underlined):
Sometimes I come across some fascinating things when trying to think outside the box for new systems. Came across this beauty today:
I have never seen a stock that has been split so far to create that high of a price. Also have never really seen such massive spikes before. But then, never really looked all that much at dying companies.
No real point to this, just found it interesting. Did a quick google search and found the reason for the last spike as follows:
from https://www.thestreet.com/story/13894736/1/dryships-soars-1-800-amid-epic-short-squeeze.html :
Fascinating how profound an impact a short squeeze can have.
I have never seen a stock that has been split so far to create that high of a price. Also have never really seen such massive spikes before. But then, never really looked all that much at dying companies.
No real point to this, just found it interesting. Did a quick google search and found the reason for the last spike as follows:
from https://www.thestreet.com/story/13894736/1/dryships-soars-1-800-amid-epic-short-squeeze.html :
Fascinating how profound an impact a short squeeze can have.
Live trades for week of 6/19/2017:
AON 6/19/2017
Entry: $136.70
EOD Exit: $136.79
Notes: Theoretical trade matches actual exactly
ABBV 6/19/2017
Entry: $70.93
TP Exit: $71.38
Notes: Partial fill only - trade not taken in theoretical system since no fill in theoretical
KSU 6/19/2017
Entry: $103.87
EOD Exit: $103.53
Notes: Delayed bar entry in theoretical system resulted in marginally lower entry and a slight discrepancy between theoretical and actual trade.
MO 6/20/2017
Entry: $77.60
EOD Exit: $77.32
Notes: Partial fill was filled completely. Theoretical trade matches actual trade exactly.
KSU 6/20/2017
Entry: $103.10
SL Exit: $102.36
Notes: Theoretical trade is a near match to actual trade
AON 6/20/2017
Entry: $136.51
SL Exit: $136.18
Notes: Theoretical trade is a near match to actual trade
ITW 6/21/2017
Entry: $149.56
SL Exit: $148.33
Notes: Theoretical trade is a near match to actual trade. A good reminder that there is no point to have a visible stop order for this system --> will fix execution to hide stop order. Net result of trade would not have changed any, so irrelevant...this time...
AON 6/21/2017
Entry: $135.89
EOD Exit: $135.30
Notes: Theoretical trade matches actual trade exactly.
RMD 6/21/2017
Entry: $78.57
TP Exit: $79.09
Notes: Theoretical trade matches actual trade exactly.
ABBV 6/22/2017
Entry: $71.41
TP Exit: $71.87
Notes: Theoretical trade matches actual trade exactly. A good end to an otherwise fairly ****ty week, but damn...left a lot on the table with that. Annoying how even wins are not satisfactory.
No trades taken in system on 6/22/2017
Current equity curve:
Couple of notes on how I am trading this now:
- Daily target risk is $200
- Risk multipliers are applied based on stock price (higher has a higher multiplier), and whether entry occurs when SPY is positive (higher multiplier) or SPY is negative (lower multiplier)
- Since it is impossible to know how many entries will be made due to using limit orders, risk is not divided equally between all entries. Earlier entries receive a larger percentage of the daily intended risk.
- The max actual risk (if all possible entries are made and all entries have the highest possible risk multiplier) is 173% the daily target risk. This is a very unlikely scenario.
Risk taken 6/19/2017 --> 97% intended
Risk taken 6/20/2017 --> 94% intended
Risk taken 6/21/2017 --> 141% intended
Risk taken 6/22/2017 --> 68% intended
Risk taken 6/23/2017 --> 0% intended (no valid trades in system)
Blue line - Actual results from live system
Red line - Theoretical results from system
Green line - Results if I were able to equally split risk between all trades each day precisely at my target risk for the day with no multipliers.
My overall thoughts so far are as follows:
- The actual trades are matching up close enough to the theoretical trades in my system that I believe it is functioning as intended.
- It is frustrating to not be able to split risk evenly between the trades accurately. I do not expect a constant divergence between the green line and blue/red lines and am for now concluding this is just due to my typical bad luck upon initiating a system. For now I remain cautiously optimistic with this system since the theoretical and actual are matching up as they should. I will do a bit of investigation into whether the limit entry is even worthwhile for the overall system if the divergence continues.
AON 6/19/2017
Entry: $136.70
EOD Exit: $136.79
Notes: Theoretical trade matches actual exactly
ABBV 6/19/2017
Entry: $70.93
TP Exit: $71.38
Notes: Partial fill only - trade not taken in theoretical system since no fill in theoretical
KSU 6/19/2017
Entry: $103.87
EOD Exit: $103.53
Notes: Delayed bar entry in theoretical system resulted in marginally lower entry and a slight discrepancy between theoretical and actual trade.
MO 6/20/2017
Entry: $77.60
EOD Exit: $77.32
Notes: Partial fill was filled completely. Theoretical trade matches actual trade exactly.
KSU 6/20/2017
Entry: $103.10
SL Exit: $102.36
Notes: Theoretical trade is a near match to actual trade
AON 6/20/2017
Entry: $136.51
SL Exit: $136.18
Notes: Theoretical trade is a near match to actual trade
ITW 6/21/2017
Entry: $149.56
SL Exit: $148.33
Notes: Theoretical trade is a near match to actual trade. A good reminder that there is no point to have a visible stop order for this system --> will fix execution to hide stop order. Net result of trade would not have changed any, so irrelevant...this time...
AON 6/21/2017
Entry: $135.89
EOD Exit: $135.30
Notes: Theoretical trade matches actual trade exactly.
RMD 6/21/2017
Entry: $78.57
TP Exit: $79.09
Notes: Theoretical trade matches actual trade exactly.
ABBV 6/22/2017
Entry: $71.41
TP Exit: $71.87
Notes: Theoretical trade matches actual trade exactly. A good end to an otherwise fairly ****ty week, but damn...left a lot on the table with that. Annoying how even wins are not satisfactory.
No trades taken in system on 6/22/2017
Current equity curve:
Couple of notes on how I am trading this now:
- Daily target risk is $200
- Risk multipliers are applied based on stock price (higher has a higher multiplier), and whether entry occurs when SPY is positive (higher multiplier) or SPY is negative (lower multiplier)
- Since it is impossible to know how many entries will be made due to using limit orders, risk is not divided equally between all entries. Earlier entries receive a larger percentage of the daily intended risk.
- The max actual risk (if all possible entries are made and all entries have the highest possible risk multiplier) is 173% the daily target risk. This is a very unlikely scenario.
Risk taken 6/19/2017 --> 97% intended
Risk taken 6/20/2017 --> 94% intended
Risk taken 6/21/2017 --> 141% intended
Risk taken 6/22/2017 --> 68% intended
Risk taken 6/23/2017 --> 0% intended (no valid trades in system)
Blue line - Actual results from live system
Red line - Theoretical results from system
Green line - Results if I were able to equally split risk between all trades each day precisely at my target risk for the day with no multipliers.
My overall thoughts so far are as follows:
- The actual trades are matching up close enough to the theoretical trades in my system that I believe it is functioning as intended.
- It is frustrating to not be able to split risk evenly between the trades accurately. I do not expect a constant divergence between the green line and blue/red lines and am for now concluding this is just due to my typical bad luck upon initiating a system. For now I remain cautiously optimistic with this system since the theoretical and actual are matching up as they should. I will do a bit of investigation into whether the limit entry is even worthwhile for the overall system if the divergence continues.
So that's pretty awesome...looks like Photobucket has changed their terms of service.
@Sharky
@timsk
@counter_violent
Any chance one of you would be able to give me the ability to edit my older posts for a couple of weeks so I can change image hosts and update the links throughout this thread? This is mostly useless to me and others (although debatable if it ever has been useful to others) without my charts.
@Sharky
@timsk
@counter_violent
Any chance one of you would be able to give me the ability to edit my older posts for a couple of weeks so I can change image hosts and update the links throughout this thread? This is mostly useless to me and others (although debatable if it ever has been useful to others) without my charts.
Sharky
Staff
- Messages
- 5,946
- Likes
- 611
foroom lluzers
Veteren member
- Messages
- 3,608
- Likes
- 138
A little while ago on this thread , you dismissed what I was saying.
In this thread you will see what I meant.
http://www.trade2win.com/boards/technical-analysis/224404-when-will-you-trade-vod.html#post2922824
In this thread you will see what I meant.
http://www.trade2win.com/boards/technical-analysis/224404-when-will-you-trade-vod.html#post2922824
You will need to be more specific than that. My recollection is that I tested what you suggested and found that the overall impact on the system as an absolute filter was not beneficial, but as a filter for position sizing was ok. I don't recall dismissing anything you posted in here...I recall testing it.
mpups
Experienced member
- Messages
- 1,022
- Likes
- 144
Hi Dr T
i was following this thread a while back but then stopped visiting the forum for a while.
how is your trading going? and where are you along the journey?
I remember you were coming up with many details about possible systems and potentially using automating software?
i was following this thread a while back but then stopped visiting the forum for a while.
how is your trading going? and where are you along the journey?
I remember you were coming up with many details about possible systems and potentially using automating software?
Last edited:
I have been testing and occasionally implementing automated systems. Haven't found anything that works all that well yet though so I am starting to look at things I have previously neglected.
Currently I am in the process of paper trading a short term (no overnight exposure) short only system. Backtests indicated somewhere around a 1.5% weekly return on average. If my paper trading results are on the order of 1% on average after about a month more testing I will bring it live on limited capital.
I am also in the process of developing a long only system with a short term timeframe but with the possibility of longer hold times. Basic idea is to limit the system to only S&P 500 stocks that pay dividends of 3% annually or greater. Enter with a fairly small target gain --> system will have no stop loss.
When I get into trades with longer hold times and large drawdowns, I will still be collecting the dividends. Being S&P 500 stocks the probability of a trade going all the way to 0 is very small.
I am just developing this now, but I suspect backtests will show some hold times exceeding a year. If I find that I only have a few of these at one time I may go ahead and bring it live. If I have all my capital tied up in these at the same time, then probably not.
mpups
Experienced member
- Messages
- 1,022
- Likes
- 144
You are very thorough with your posts/journal thanks for all that effort its very impressive in itself! I've been having a skim back through them.
did you achieve this or more yet? had many positive months or longer periods?
it took me about 9 months to have a positive month, and about 4yrs to have a positive 12months !
RachasfromPortugal
Junior member
- Messages
- 33
- Likes
- 1
no words...
