What is a decent sample size?

graemenash

graemenash

Active member
Messages
144
Likes
18
When testing a system, what kind of sample size would you look for it to be profitable over before you had confidence in it as an "edge"?
 
A good few hundred! I'd like to experience a drawdown period, too, so you'll know whether you could survive if you were trading a big size.
 
From a viewpoint of theory, might be worth starting with the central limit theorem etc...
 
I want to see a test that encompasses at least one full market cycle - uptrends, downtrends, consolidations; low volatility and high volatility - all for whatever timeframe is being contemplated. How many trades that is depends on the system's trade frequency.
 
Get as much data as you can get, 10 years or more. Then split it 70/30.

Backtesting on 70%

Forward testing on 30%

The DRAWDOWN IS KEY... As low as possible...
 
20-25 trades is more than enough to test out the edge for a method. you should look not only at the win rate but also the risks attached to each trade. if you ignore either of those you could end up in a situation with a 70% win rate but a risk on average so large the 30% of losers equals a net loss. it is the utmost importance you take each trade as disciplined as you can to ensure correct results.


good:
take another sample size. if outcome is the same or relative then this method is ready to be tracked as part of your trading toolbox ( but for at least 6 months to a year maintain records for this method to ensure longevity)

bad
only after the sample size is complete should you look where things went wrong and make a changes for another sample to be taken. continue in this fashion as long as the sample results improve but if you get 3 bad samples in a row then move on.
 
This is a mechanical edge I'm testing, traded automatically - the trader has nothing to do with it in this case lol :)
 
graemenash said:
I don't "get" most of your posts to be honest, but still get along fine :LOL:
Click to expand...

I'm glad. Like someone else used to say, I post for the absolute minority. Your post is typical of many others. You come here asking rudimentary, school boy questions then when you are encouraged to face reality you ignore it and say you are "getting along fine". :rolleyes:
 
new_trader said:
You come here asking rudimentary, school boy questions
Click to expand...
Last time I checked, this is a discussion forum where traders seek the opinions of other traders...

new_trader said:
say you are "getting along fine"
Click to expand...
Mate, I've been trading for a living for years; but if it helps you sleep at night to think you're somehow better than me then go ahead.

I'm not going to bother replying to any more of your posts, so I'll even let you have the last word if you want... treat yourself :p
 
new_trader said:
I'm glad. Like someone else used to say, I post for the absolute minority. Your post is typical of many others. You come here asking rudimentary, school boy questions then when you are encouraged to face reality you ignore it and say you are "getting along fine". :rolleyes:
Click to expand...

Stop using that purple text and trying to be someone your not, its just sad. You don't post for the minority because the minority already knows far more than you and can learn nothing from you sarcastic and bitter remarks.



I am under no doubt that mechanical systems can work, but I think that in the vast majority of cases it is the trader that gives any system/method an edge. It winds me up when people generalise, especially about trading because it is really a subject than can not be generalised in any way.
 
graemenash said:
When testing a system, what kind of sample size would you look for it to be profitable over before you had confidence in it as an "edge"?
Click to expand...

The more the better.

I like to test several thousand instances... i have about 5,000 across 10 different markets going back many years.

1000 is the barest minimum i would need before i start feeling confident with my edge.


Bill Eckhardt: I know of no way to validate conjectures concerning technical trading without back testing; however, this procedure is fraught with peril--we all know horror stories. Having adequate amounts of data for reliable inferences is only one of many problems facing the technical analyst, but it is as crucial as any. Statisticians tend to consider that more than about 30 instances constitutes a large sample statistic. For futures price research this is a recipe for disaster. The underlying probability distributions in this subject are so exotic and pathologic that those subtle techniques that statisticians use to squeeze significance out of sparse data are all decidedly out of place.

To make even moderately reliable judgments about a kind of trade, you need something like 300 instances. This is a minimum figure. I don't feel comfortable acting on research results unless I have several thousand instances.
Click to expand...
 
I don't believe back testing has much value at all. The fact is, markets change and that much should be obvious since last autumn when things have really altered. The further back you back test the more likely it is to be different from the present conditions.

Apart from that, back testing really always ends up curve fitting to try and get a profit over the period tested. Even in totally random charts, there will always be patterns but they are very unlikely to repeat again in the future.

If you dispute this, just think how many hundreds of thousands of people have back tested millions of 'systems' and yet we still don't have one proven to make consistent profits.

Still people waste countless hours on the dream instead of actually learning to trade - which requires discretion and changing your method to suit the chart of the hour. Or, of course, you could always get an EA and get rich while you sleep...
 
graemenash said:
Agreed, I've lost count of the amount of "holy grails" I've discovered while back-testing :)

It's purely a forward test that I'm focusing on.
Click to expand...


IGNORE the top 10% of trades (the best ones) and see whats left. Often an Eye opener I find.
 
i am no believer in automated mechanical systems. maybe its because if they were very good there wouldnt be trading desks in banks and funds filled with traders. i think in a world where such a concept was real you would have massive funds being run by a few computers in some guys spare room. i do believe they could produce a tiny return over time but it would need to be based on some very high level processing calculations that the average joe couldn't easily achieve. i have heard stories of some banks having million dollar systems that turn over a % return but as for consistency your guess is as good as mine.. anyways i personally love the art of charting and reading price structure but i dont use indicators so its more enjoyable for me.. well my 2 cents anyways and i am sure this might conjure up a debate of claims of a working automated systems out there. i am not ar5ed to argue about this but i would like to hear opinions of people who have tried or are trying
 
You must log in or register to reply here.

Similar threads

M
Sample Size
Replies
7
Views
3K
forex99
F
hhiusa
Statistics question
Replies
19
Views
4K
hhiusa
hhiusa
L
Sample size?
Replies
9
Views
2K
LiamH
L
robster970
Can anyone explain this contradiction?
Replies
6
Views
2K
robster970
robster970
T
When to pull money on the line??
Replies
10
Views
3K
BroadSword
BroadSword
Top