Backtesting assurance

[Strangelydifferent]

Dear all

I have been trying to learn to use a backtester for a while and also developed indicators on the platform that I use that attempt to perform an independent assessment of the fraction of times a particular set-up leads to success.

What strikes me as clear is that (for swing trading stocks by the way) high success probabilities are very stock-dependent and generally last for a few months only. After that the market moves on a bit and I have to try new settings, new set ups completely or try different stocks to find high success rates.

So, where this leads me is that at any time my assessment of success probability is often based only on ten or so signals over a couple of months. This contrasts markedly with advice I have read on T2W forums that I should be looking for success of a set-up to be demonstrated over several years of back testing.

It might be I am simply not trying the right set ups. Any thoughts please?
Many thanks in advance

 

[terrylee]

Dear all

I have been trying to learn to use a backtester for a while and also developed indicators on the platform that I use that attempt to perform an independent assessment of the fraction of times a particular set-up leads to success.

What strikes me as clear is that (for swing trading stocks by the way) high success probabilities are very stock-dependent and generally last for a few months only. After that the market moves on a bit and I have to try new settings, new set ups completely or try different stocks to find high success rates.

So, where this leads me is that at any time my assessment of success probability is often based only on ten or so signals over a couple of months. This contrasts markedly with advice I have read on T2W forums that I should be looking for success of a set-up to be demonstrated over several years of back testing.

It might be I am simply not trying the right set ups. Any thoughts please?
Many thanks in advance

To me back testing is flawed. Learning from nonstationary market data streams finds an average that is not representing future distributions very well. It took me several years to realize it.

 

[Strangelydifferent]

Thank you - other views are welcome but that brings me to my next question already:

If backtesting provides "limited assurance", then how does a trader know that they have "an edge" (to quote a phrase I have seen on T2W a lot) or equivalently in my language "a basis for making decisions"?

i.e. What other assurance can I consider, aside from blind faith?

 

[jm99]

there are no assurances. the best u can do is backtest and walk forward out of sample (this is the important part). if these terms do not mean much to you then look at pardo for techniques on how to do this, as the process can be seriously flawed if not done correctly due to the numerous biases we are prone 2.

even then there is no guarantee that the method/system will not stop working. u just have to embrace the uncertainty and limit risk via position sizing and diversification when possible

 

[Benton D Struckcheon]

You would know if it's a reasonable basis for a decision by whether you can take the same method and use it with some other, related trading instrument.
And, if you have to tweak it somehow, you should be able to explain to yourself why.
I use options action to see whether things are bullish or bearish, but this differs with what you're looking at. I'm in the US, so I look, for instance, at options on SPY, which is the ETF for the SP500. I also have recently been looking at options for GLD, the ETF over here for gold. I use the ratio of puts to volume for SPY (this isn't the same thing as the put/call ratio, but it's close. I find this works better though), and assume they're being used for hedging, meaning that more puts is not necessarily bearish. But with GLD, I use the ratio of calls to volume, because the action there shows that the options are being used for speculation, not hedging, and speculators tend to buy calls.
So different approaches, but they're based on how things happen in real life.
Also, some rules of how the market works:

1 - At extremes, there will be a squeeze in the opposite direction.
2 - Markets spend more times rangebound than trending, so you have to account for that.
3 - If you have a rule for how the market behaves on the way up, it may not work the same way for a downtrend. Down is different, and usually faster, than up. This seriously complicates things.
4 - Use the 2nd derivative for measuring if a trend is accelerating. (I've never seen anyone say this, but I'm sure traders use this, and I've found it to be very insightful.)
5 - A system is only a guideline. Real life will vary. To paraphrase the Great Communicator, your approach to your system or anyone else's should be "Trust but hedge".

 

[Strangelydifferent]

Hi

JM99 - Thankyou - I think I see the overall point but actually I haven't heard the phrase "walk forward out of sample" before. Also I'm not familiar with pardo though I have looked him up elsewhere. Amazing what you find if you only have a starting place...I'll check it out

Benton - Thank you too, let me check if I get your first point. Do you mean that, assuming I have several decision criteria that works "for some short period of time" that I might be most confident in those that work most often for different securities?
Your option points are interesting but where do you get the confidence to assume that "puts to volume" indicates hedging whilst calls to volume indicates speculation?
I think I follow each of your latter "rules", I had become aware of 1 and 2, loosely. I'd not noticed 3 todate but it sounds worth looking out for. So let me just check - I can construct an indicator (not one that gives buy signals etc) just to evaluate how much your point 3 is happening. I can also do this to determine 4. At the moment I see the MACD histogram as an example of acceleration. Are there any other common measures? So, I can choose to look at this at any scale to provoke thoughts about what is happening to trends. 5. Hedging strikes me as a superficially easy but actually rather elusive concept when I try to think it through.

So, I think I need to get my head down!

On a different subject, point 5: hedging strikes me as a novice as a zero sum, less commission. The reason I'm stuck here is as follows:
a) if I try to buy two stocks that, say, might move in different directions if some unexpected news comes in (say a change in the oil price) then
b) if any divergence does occur I keep the one that benefits and cut the other in accordance with stop losses.
c) I now have a single un-hedged stock and a loss from the previous sale hopefully mitigated by the gain in the one that I have kept.
i.e. I'm back where I started. Any thoughts?