Backtesting - is it a valid concept ?

[Glenn]

I wonder what others think about the concept of backtesting ?
One the one hand it perports to find the best set of parameters for a given method and thereby tune it to the period tested.
If the results are good, then these parameters can then be used with some (?) confidence for live trades.
On the other hand one could argue that backtesting is just curve fitting and therefore simply 20:20 hindsight.

Seems to me that for a backtest to have any validity, it must produce a smoothly increasing equity (profit) curve with no wild girations. This at least demonstrates that the method and parameters found worked fairly consistently over the period tested.
Of course a lot depends on the period(s) selected.
Again to have any validity, I think that a test must either be run over a considerable time period, embracing many different market conditions and yet still remaining consistent, or it must be run on selected market conditions with a view to using the optimised parameters from the test in that type of market.

To disprove the theory of curve fitting, one suggestion is to take a period of time, say 5 years, and run your test over the first half of the data to get some results.
These results can then be run over the second half of the data as if it were the unknown future. i.e. a kind of Forward Testing.

One thing's for sure, the Software vendors must love things like backtesting. What an opportunity to develop and sell a nice technical product, using the math co-processor and the disk drive of your PC to the full. A nice 'gadget'.
Lovely jubbly - at least for the vendors.

Enough from me. What do others think? In particular anyone who has used a backtested system in the real market sucessfully.

Glenn

 

[Grey1]

Glenn,

Before you ask about back testing being of any value , you have to ask yourself another more important question..
DOES PAST CORRELATE tO FUTURE ?

You can only have three answers .


1) yes it does..
2) No it does not
3) it does to some degree
if you think number 3 is the answer then your back testing will be effective to the same degree ..

 

[Rognvald]

One more question
"and for how long" which I suppose is really a subset of "some degree"

 

[Glenn]

Grey1
"DOES PAST CORRELATE tO FUTURE ?"

Agree the question, hence my thought about Forward Testing as a means of finding out.
But are you suggesting that one's degree of belief in backtesting is only a guarantee of good backtests, rather than good future results ?

Rognvald
"and for how long" ?
Again I'm thinking that forward testing may help to determine this to some degree i.e. within the bounds of available historical data.

One way or another most traders are looking for something which works enough of the time to enable money management to produce a profit.
So how do they find something which works 'enough of the time' ?
And yes, how will they know when it stops working (apart from the obvious) ?

Glenn

 

[JonnyT]

Hi Glenn,

Solve that and you've solved the worlds problems.

In short there is no answer.

JonnyT

 

[Grey1]

Glen,

Quote " are you suggesting that one's degree of belief in back testing is only a guarantee of good backtests, rather than good future results ?"

yes Glen, you are right,, people go to a huge length to back test a strategy and optimise various parameters such as Win/Loss ratio, Sharp Ratio, net profit not thinking to what degree the result of the back tested can be trusted for future performance,.

If you forward test you soon be disappointed .. In fact you soon realise that very best systems donot give you more than few % edge if forward tested.. This edge is often gets over ridden by the market costs ... This is why Trading is such a tough game..

 

[barjon]

Glenn,

Before you ask about back testing being of any value , you have to ask yourself another more important question..
DOES PAST CORRELATE tO FUTURE ?

You can only have three answers .


1) yes it does..
2) No it does not
3) it does to some degree
if you think number 3 is the answer then your back testing will be effective to the same degree ..

Grey

There is one lesson of history that will, with certainty, repeat in
the future - if demand outstrips supply, prices will rise and vice
versa.

Some ta, broken support becoming resistance for example, has an
underlying supply & demand logic that is both sound and readily
identifiable. However, in a lot of ta (and indicators in particular)
it is extremely difficult to identify underlying supply & demand
logic (that's not to say it's not there tho' - it might just be very
obscure ).

imo if your method has an identifiable and sound supply &
demand logic supporting it and it has worked when backtested,
then the chance of it repeating in the future is quite bright. If you
can't find the logic then, however well it's worked in the past, the
chance is just a chance.

good trading

jon

 

[Grey1]

barjon

Lets say British Airways Q1 is greater than expecation in year 2003 and as a result the demand increases and price goes UP

Now what would you say BA shares would do if Q1 in 2004 is even better.... Will the price go UP or DOWN ?

What a dilemma ?

 

[barjon]

grey

go on, you're kidding me you're not really interested in WHY
it's in increased demand are you?

your chart will be telling you if demand is increasing and the price
moving - all you've got to decide is when and if you're going to
take advantage of it. Course your chart may also be telling you
that the supply is going to increase quite soon, so you may wait
to take advantage of that.

Still a dilemma of course :cheesy: :cheesy:

jon

 

[Grey1]

Barjon,

Quote "go on, you're kidding me you're not really interested in WHY
it's in increased demand are you? "

No I am not but I am intrested in your answer for Q1 2004 do you think the price will go up or down ..?

 

[Rognvald]

Price includes all factors known to the market and these include expectation. Therefor if the expectation is that they will do as well year on year and results are in line the price will be largely unchanged as that factor is built into the price. If the results exceed expectation there will be a rise - if less then a fall (all other things being equal)

 

[Grey1]

Rog,

INTC had a better than expectation result out and the price fell.. Stock B can have a lower expectation and market might see it as a good news..

ONE IS NOT CLEVER ENOUGH TO KNOW THE EFFECT OF NEWS ON STOCK PRICE

The BA example I gave was on 2000-1999 when Q1 result was bad and price increased while there was an optimistic earning report for Q3 and the price fell sharply..

 

[barjon]

grey

sorry I thought it didn't matter! It might go up or it might go down
whatever it does the chart will say what's happening.

Absolutely agree with your in bold capitals reply to Rog

jon

 

[Grey1]

Agree barjon,

Like any thing else this phenomana has been studied by few researcher to gain an edge in the market and still no one has come up with any solution why mass behaves differently under the same circumstances how ever the post event analysis of the reaction to comets is easily explianed i nterms of supply and demand..

Now my point in the back testing question was to demonstrate that the back tested result of a strategy only to some degree correlates to the future .. The correlation is not a strong but a loose type .. ( I have avoided to use the term "of the Hinged type .. This gives me the room to defend technical Analysis "


look at this link

http://ideas.repec.org/p/nbr/nberwo/7330.html

 

[barjon]

grey

I hadn't meant to give the impression that technical analysis
needs defending. It's just that I find some of it (maybe the
curve fitting variety you talked about earlier) to be a bit iffy when
the underlying supply & demand logic is not apparent.

I think we're on the same side?

jon

 

[Rognvald]

I didn't say I was - only that if there is a gap between perceived value and price then the price will change accordingly. Expectation/sentiment is a substantial element of price. That is my opinion - it may not be yours.

 

[Glenn]

jon
The supply demand situation you describe is similar to the analysis of PnF charts.
e.g. double and triple tops and bottoms etc.
Dorsey provided stats on the success rate of various such patterns in his first edition of Point and Figure Charting. Note these have been withdrawn from his later edition.

Not wishing to go off on too much of a tangent here but I believe he didn't adhere to his own medicine i.e. Long patterns work better in bullish markets, so use them in those conditions, and vice versa. Going long on a breakout in a bear market is somewhat risky etc.
So looking at an individual stock chart is not enough. The prevailing market at any time has a great deal of influence.
"75% of the risk in any stock is in the market and sector" - Dorsey.

Grey1
Agree the problem of market costs. Suggests aiming for longer term positions, larger moves, fewer trades.... (and consequently needing bigger b*lls.)
As regards correlation of test results to the future, I think there is a danger in looking at individual stock charts for the reasons I gave above.
Indexes are a different matter because they tend to encapsulate most of the sentiment at any time. But even they can be swung by large caps, a rush into/out of techs or utilities etc.

I suppose I'm saying that backtesting needs to have more than one layer in order to take account of the available information.
You can test drive a car with bald tyres in the dry and get consistent results. You buy the car and then it rains........
Glenn

 

[Grey1]

Quote "You can test drive a car with bald tyres in the dry and get consistent results. You buy the car and then it rains........ "

LOL.. :cheesy: :cheesy: :cheesy: :cheesy:

 

[BBB]

I think time frame has a lot to do with backtesting validity.

If you are backtesting an intra day system the results would probably be less reliable due to the manipulation that goes on. A daily or weekly system may have more valid results as price is harder to manipulate over the longer time frame, and therefor you get a better picture of the true demand and supply and the sentiment/participants reactions to the market conditions.

I have never tested this idea, its just my feeling (personally, I don't care much for backtesting at all due to my trading style).

 

[mr_cassandra]

A couple observations

Basically everything everyone you know is doing has been derived from a back-test of some sort, whether they realize it or not.

That being said, from my experience with my own system, I’ve found the following to be true of back-testing.

1. It only proves that your method worked during the time you checked and only for the market events which occurred during ‘that’ time’
2. The longer the period of the back-test, the better. It would be great if it covered enough market types to include a bull market, flat market and a bear market.
3. I’m only a few cycles into real-time trading with my systems version 21 and can tell you from experience that what has derailed each prior version was an event or market anamoly which was ‘not’ programmed in and/or occurred in the prior several years back-test.

I wonder what others think about the concept of backtesting ?
One the one hand it perports to find the best set of parameters for a given method and thereby tune it to the period tested.
If the results are good, then these parameters can then be used with some (?) confidence for live trades.
On the other hand one could argue that backtesting is just curve fitting and therefore simply 20:20 hindsight.

Seems to me that for a backtest to have any validity, it must produce a smoothly increasing equity (profit) curve with no wild girations. This at least demonstrates that the method and parameters found worked fairly consistently over the period tested.
Of course a lot depends on the period(s) selected.
Again to have any validity, I think that a test must either be run over a considerable time period, embracing many different market conditions and yet still remaining consistent, or it must be run on selected market conditions with a view to using the optimised parameters from the test in that type of market.

To disprove the theory of curve fitting, one suggestion is to take a period of time, say 5 years, and run your test over the first half of the data to get some results.
These results can then be run over the second half of the data as if it were the unknown future. i.e. a kind of Forward Testing.

One thing's for sure, the Software vendors must love things like backtesting. What an opportunity to develop and sell a nice technical product, using the math co-processor and the disk drive of your PC to the full. A nice 'gadget'.
Lovely jubbly - at least for the vendors.

Enough from me. What do others think? In particular anyone who has used a backtested system in the real market sucessfully.

Glenn