disktrading.com intraday data

I'm planning to get futures and forex data from disktrading.com to backtest some trading systems on hourly data.

There's a couple of things I'm concerned about.

If I order the whole set of historical data for futures, how do they construct the continuous contracts that are mentioned. There doesn't seem to be any explanation on their site and I would ideally prefer to define the rollover date myself.

What is the error count like? Are there many problems, or rather how frequently do they occur?

I guess I need to contact disktrading.com myself, but a couple more questions if anyone feels like being their support line:

do you have to request the specific timeframe, or do you get the raw ticks and some software to convert it yourself?

I am satisfied with them. I've been using them for years: cheap and good data as far as I am concerned. Yavor is the owner and he's quite friendly and honest. He'll explain to you whatever questions you ask him. I can't answer your questions, because yes, they're continuous, but I don't know how they did it. Also, you could ask weighbridge. He bought the whole package for a bit over 100 dollars. He seemed satisfied as well. I don't know about the error count. There's definitely some holes, but they may be due to holidays and such. I think that's the case.

Oh, and about the tick data and everything, don't worry. You can test this: for 20 dollars you buy one symbol, like the EUR, or GBP, and you'll see that you'll get the whole data set, from daily up to tick data. All timeframes. They're really great in my opinion. Great organization, and order. Efficient. I can't swear about the quality, but I've compared them to other sources, like IB data (which I get for free, since I use IB), and anfutures.com data, and they're pretty much the same. My systems make the same money with all this data. True: not exactly the same money, but almost exactly the same money, like closer than 95% to one another.

They even worry to give you updates for a while, like probably a year or so. I never worried about getting my updates so I couldn't tell you exactly, but the updates (contract by contract) were there for about a year or so.

I don't want to offend anyone on here, but I was quite disappointed from disktrading. I bought the historical data for the YM, which costed 20$. But after downloading them, fitting the time to my time zone and comparing them with other data feeds, there were huge differences. Several 12 hours (!) bars were completly different from the real data, sometimes they went even into the wrong direction.
Perhaps I did something wrong, but for me, the data was quite useless. I would first buy one symbol and check the data. If all is okay, you can buy the rest from the package.

Regards,

Sepp

Yeah, good idea.

Master_Sepp said:

Perhaps I did something wrong, but for me, the data was quite useless.

Click to expand...

It wasn't you, I have had similiar issues with disktrading. They were a poor data feed back in 2000, but had an edge with pricing. Today, many more vendors exist for intraday data with simliar pricing. I would just look elsewhere.

While you're at it, can you please give us the name of one of these better data vendors? If possible the best one (of the non-expensive ones). This is precious information. I thought they were great, but if you tell me a better one, I'll be glad to switch.

Sepp, Kip -

did you contact Disktrading's customer support? Presumably they have customer support. If so, what did they say?

I appreciate your feedback and will certainly bear what you say in mind, but I can't really give much weight to what you say unless you tell me what Disktrading's response was.

I have actually had constant problems over the last year with my current data supplier and I always go to the supplier with the problem. Often to my embarrassment it is my own mistake, not theirs.

kipptrader said:

...many more vendors exist for intraday data with simliar pricing. I would just look elsewhere.

Click to expand...

Look where exactly? Any names? Otherwise you're not helping us very much.

Adamus said:

Sepp, Kip -

did you contact Disktrading's customer support? Presumably they have customer support. If so, what did they say?

I appreciate your feedback and will certainly bear what you say in mind, but I can't really give much weight to what you say unless you tell me what Disktrading's response was.

I have actually had constant problems over the last year with my current data supplier and I always go to the supplier with the problem. Often to my embarrassment it is my own mistake, not theirs.

Click to expand...

When you work with data as long as I have, you can just tell when something isn't right with the source. But to be sure, I have made it my practice to run a statistical comparison on new sources using overlapping symbols I have in my archive.

I use a variety of vendors as different vendors offer unique markets and/or duration of history. No historical source data is perfect just like no real-time quote feed is 100% accurate. But you want to see a good high 90s (98%+) in reliability.

So let's start with tickdata.com as our primary source. Again, no data is perfect, but tickdata is very reliable but also very expensive. I will then run a correlation between my primary source and another source.

In the case of Disktrading, the correlation was just plain pathetic. From my past comparision of the data, it looked patched; using multiple sources over time to patch together a price history. As Sepp has said, the data is wild: too many gaps, wild price swings, and erratic volume.

The pros of Disktrading of course is price and they do offer a large selection of futures markets. The cons are accuracy and the data not being clean.

My opinion is just that, my opinion. So check it out for yourself. $20 for a single symbol isn't must to run a test.

travis said:

Look where exactly? Any names? Otherwise you're not helping us very much.

Click to expand...

Before selecting a vendor, you need to narrow down what you are trading: Furtures, forex, stocks, u.s., europe, asia, etc.... Every vendor covers a subset of markets, so what you trade or back-test is important in finding a vendor.

If your want my opinion, let me know what you are looking for first, and I can give you a list of places to try.

Thanks. This is good information. Here's the details of my 2 questions:

Question # 1:
I'd like to know where I can buy data for these CME Group futures: EUR, GBP, JPY, ES, YM, CL, GC, ZN. Obviously, for the same price as disktrading.com data but of a better quality, as you said in your earlier post:

...many more vendors exist for intraday data with simliar pricing. I would just look elsewhere.

Click to expand...

Question # 2:

kipptrader said:

[...]
But you want to see a good high 90s (98%+) in reliability.
[...]
In the case of Disktrading, the correlation was just plain pathetic. From my past comparision of the data, it looked patched; using multiple sources over time to patch together a price history. As Sepp has said, the data is wild: too many gaps, wild price swings, and erratic volume.
[...]

Click to expand...

Thanks, this is very good and precise information. What would you say the percentage rating of the disktrading.com data reliability if you exclude volume (I don't need volume)? The reason I am so interested in knowing how good they are is that I based all of my back-testing on their data (and so far the forward-testing is doing just as well as the back-testing, which would lead me to believe their data is good).

kipptrader said:

My opinion is just that, my opinion. So check it out for yourself. $20 for a single symbol isn't must to run a test.

Click to expand...

OK sure. But this will not be a comparative test, obviously - I'd need to splash out a couple of grand at tickdata.com to do that. However I'll take your word for it that there are issues with disktrading.com's data, although whether that has any impact on any trading system I'm likely to come up with, I don't know.

One way of looking at it is that mistakes in the backtesting due to bad data will even themselves out.

Of course I'd rather have perfect data, but the money to pay for it would come out of my trading capital. Whether that's a good trade-off or not, I doubt, my account size being relatively modest.

I would not get too hung up about data quality for use with backtesting. Remember that backtesting is only ever an approximation, because you can never know what price you will get when you trade for real. There are some people who believe that all backtesting is a waste of time.

Personally I believe it does have value for two reasons: firstly it gives you an indication of the likelihood of success with the system, but secondly and more importantly, it allows you to discover conditions which you may not have thought about when designing the system.

I would endorse disktrading's data. I have used it for a couple of years. I like the price and the range of data, and as someone said earlier, you can always email the company owner and get a friendly response. As for its accuracy - no idea, but it suits my backtesting needs.

OK, I'm pretty satisfied with that point of view, not having the funds or the time to get hold of more expensive data to compare. Thanks.

It brings me back to my other question now: how are the futures continuous contracts put together? Hopefully there aren't any huge gaps at roll-over, because that would actually cause me concern.

I think I can help with that question as well. When you buy data from them, you get a long email explaining various things. Below is an extract that talks about how he puts together the continuous futures files:

Futures data in our database is in continuous format - there is one file per symbol named after the currently most active contract and containing data from the previous contracts for the time they were most active. We do this by always collecting data for the most active contract in one file and closely comparing the trading volume to the volume of the forthcoming contract. When the new contract becomes more active than the current one, we "rollover" by re-naming the file after the new monthly contract and resuming data collection for the new one, because it is now most active.

Sounds good, although there are some markets which have contract months that you wouldn't want included in the continuous contract due to lack of volume like Crude, or other markets where you wouldn't want to trade the spot month, e.g. Eurodollars where directional trading is better on the 3rd contract out.

travis said:

Thanks. This is good information. Here's the details of my 2 questions:

Question # 1:
I'd like to know where I can buy data for these CME Group futures: EUR, GBP, JPY, ES, YM, CL, GC, ZN. Obviously, for the same price as disktrading.com data but of a better quality, as you said in your earlier post:


Question # 2:


Thanks, this is very good and precise information. What would you say the percentage rating of the disktrading.com data reliability if you exclude volume (I don't need volume)? The reason I am so interested in knowing how good they are is that I based all of my back-testing on their data (and so far the forward-testing is doing just as well as the back-testing, which would lead me to believe their data is good).

Click to expand...

kipptrader, I am still waiting for an answer to my questions. Otherwise, what's the use of telling us there's better data vendors than disktrading (at the same price) and then not giving us any names? For now the only information you gave us is:

...many more vendors exist for intraday data with simliar pricing. I would just look elsewhere.

Click to expand...

What is going to take to get one link?

travis said:

kipptrader, I am still waiting for an answer to my questions. Otherwise, what's the use of telling us there's better data vendors than disktrading (at the same price) and then not giving us any names? For now the only information you gave us is:

What is going to take to get one link?

Click to expand...

Check the data resources at http://www.quantitativetradingsystems.com/resources.html#dataproviders under the Historical Data section or at http://coreyhoffstein.com/automated-trading-resources/ under Historic Data.

I have used personally or for work just about every feed around, so if you have specific question about any feed, let me know.

Oh good... finally. Thank you. Of course at this point, I had bought most of the data (CAD, AUD, CHF, NG, SB) from Disktrading.com... and actually I was satisfied for all futures except two of them which they don't have (only for 5 hours instead of 20 hours a day): Corn and Soybean. Initially I had problems with other symbols, but they wrote me and sent me the good data, and now I will use your great links to try and find a good data vendor for just those two symbols, ECBOT's Corn and Soybean futures.

One more thing about disktrading.com. For having paid them 50 dollars for those few symbols I complained about (with other companies it would have cost me thousands of dollars), they were extremely helpful: they sent me several emails, explaining me what the problem was (they thought I wanted the forex data), sending me the futures data for those three currencies, and offering a refund for Corn and Soybean (because they don't have the data after the open outcry hours).

So far the other data vendors are selling for 1000 dollars what disktrading.com is selling for 20 dollars. There's really no comparison. No quality could justify such a difference.

OK so I downloaded all the omz files from disktrading for the 10 forex pairs I bought. Once I've got them into TS, what kind of sanity checks do I need to do on the data?

I guess spikes in the data are the main worry, although missing periods would also be a concern if they are extended, especially if in peak trading hours.

Not sure I have time to check all that though - I'm flying by the seat of my pants now.