NO! The only way is to read off the data from the chart or table or whatever by eye. 2nd choice is see if the program allows you to save or export the data to a file. Last alternative is find out if the program supports some kind of interface (e.g. DDE, programmable .. etc) and implement or use that.
IF you can use your mouse to hilight and copy the data on the screen (so you can copy it into another program), then you should be able to do this automatically using a macro program. This is never as good as using a real api, or even a dde link, but it can work.
Google for "screen scraping" for more info.
Steve
Google for "screen scraping" for more info.
Steve
blackcab
Established member
- Messages
- 523
- Likes
- 51
The answer is yes, you can do this, but maybe no-one's written the program yet. A program could request a chart from a Web site, decompress the GIF/PNG into a flat format, then 'read' the scale, bars, time by knowing what colour pixels to look for, what shape the characters are, etc. It would be a long, tedious program to write but a fun challenge if you're into that sort of thing. If you had such a program, you could schedule it to get charts of any instrument every day and have free, up to date data.
Strategic Trader
Active member
- Messages
- 100
- Likes
- 2
you can read any data that you see on a screen - either by writing a program that directly reads the screen data or else 'sniiffing' the data sent to the screen, so you can effectivly automate programs which seemingly are only manually controllable
TWI
Senior member
- Messages
- 2,559
- Likes
- 269
When I was working at Enron we had some code that would scrape data from websites each day and dump it into our own database. So this definitely can be done. An example of the kind of data we "scraped" is
http://www.eex.de/
http://www.eex.de/
Strategic Trader
Active member
- Messages
- 100
- Likes
- 2
scraping data from websites is a little different - since you then have the internet pages that can be directly read in code - but you can also just have a program that can read any data that you see on a screen, regardless of how it is originated
The answer is as stated by some, YES, but the ease with which it can be done depends entirely on the delivery method. Such items as graphs may be downloaded seperately from the web page and displayed on screen. The html when downloaded does not usually include the graphics but can be read as a text file and data extracted to datafile including csv etc. The problems only really begin when trying to scrape flash screens but even then the problems are not insurmountable, but, in all cases it requires a small amount of programming effort. All of the above can be achieved with AUTOHOTKEY. This site is updated on a very regular basis and the author responds very quickly to forum users requests. The members are extremely helpful to newbies. This environment has worked well for me for the last 12 months.Anyone who has used/or prefers basic could use AUTOIT3 from which AUTOHOTKEY was forked. The language recently underwent major updates to make it state of the art BASIC including calls to DLL's.It is a compileable environment annnnnnnd its FREE !
QED.
QED.
Strategic Trader said:
Interested in how you go about doing this - in the good old days, you'd just read the data straight out of the memory mapped screen data - is this still possible under Windows with the HAL* in the way? Or do/did you read something to read from memory on the graphics card?
(*Hardware Abstraction Layer)
@zzaxx99
Basically unlike Html which can downloaded and scraped in background, flash presentations are more easily done on screen. Some screens will allow CtrlA CtrlC sequences to be pushed onto the clipboard where they can be retrieved as text and proccessed accordingly. However some screens reject this sequence,but if the window has an edit control no probs, just Alt Edit Select All - AltEdit Select Copy again all details will be pushed to the clipboard. It is important to make sure that the box to be trapped is left clicked before above operations. Because this method is interactive, it really needs to be run on a machine on its own,but then several screens can be trapped in sequence.I use AUTOHOTKEY to automate the process including proccessing of the clipboard data into realtrime csv data files . (Whoops now the secrets out of the bag)
QED.
Basically unlike Html which can downloaded and scraped in background, flash presentations are more easily done on screen. Some screens will allow CtrlA CtrlC sequences to be pushed onto the clipboard where they can be retrieved as text and proccessed accordingly. However some screens reject this sequence,but if the window has an edit control no probs, just Alt Edit Select All - AltEdit Select Copy again all details will be pushed to the clipboard. It is important to make sure that the box to be trapped is left clicked before above operations. Because this method is interactive, it really needs to be run on a machine on its own,but then several screens can be trapped in sequence.I use AUTOHOTKEY to automate the process including proccessing of the clipboard data into realtrime csv data files . (Whoops now the secrets out of the bag)
QED.
Can't see that this will work in my case - I want to be able to grab streaming prices off Advfn or MoneyAM and convert them to CSV - cut & paste doesn't work (presumably because all of the prices are being displayed by a Java applet).
In theory, you could capture the screen buffer and decode it, or maybe grab the window context and interrogate it, or patch the Java runtime to intercept the screen calls, or install a proxy server to intercept it - be any awful lot easier if someone else has already written it, though!
In theory, you could capture the screen buffer and decode it, or maybe grab the window context and interrogate it, or patch the Java runtime to intercept the screen calls, or install a proxy server to intercept it - be any awful lot easier if someone else has already written it, though!
zzaxx99 said:Can't see that this will work in my case - I want to be able to grab streaming prices off Advfn or MoneyAM and convert them to CSV - cut & paste doesn't work (presumably because all of the prices are being displayed by a Java applet).
In theory, you could capture the screen buffer and decode it, or maybe grab the window context and interrogate it, or patch the Java runtime to intercept the screen calls, or install a proxy server to intercept it - be any awful lot easier if someone else has already written it, though!
I think what you need is OCR software that can be automatically used to grab a screen print, recognise the characters and then place the text elsewhere so it can be analysed. I did some work on this back in my Deal4Free days when they often took out the stops I had placed with them even though the stop price was never reached in the cash market. I wanted to write a program that would scrape D4F's prices off the screen, and automatically exit my position (or at least alert me) when my stop was reached without D4F ever knowing where my stop was. I never finished it as I gave up on discretionary trading - too much like hard work.
The name of the software I looked at is Textract. It comes in both DLL format which can be called from most programming languages, and command executable format which can be scripted. You tell it the screen coordinates you want to capture (in your case where the ADFN prices are displayed) in x and y values, and it will take a screen print, OCR it and put the text in a file/variable ready to be used. Put it in a loop and you have a solution.
Cheers,
Brendan
@bdholmes
Yes , that was my next topic, but you seem to have covered it well enough. Bit more hard work though unless your in to serious programming. Worth a look might be Ibasic, very easy environment to work with, Oh and thanx for the tip on textract, have'nt seen that one before.
Yes , that was my next topic, but you seem to have covered it well enough. Bit more hard work though unless your in to serious programming. Worth a look might be Ibasic, very easy environment to work with, Oh and thanx for the tip on textract, have'nt seen that one before.
2005-03-11 02:35 London
@zzaxx99 :
Go here www.structurise.com/kleptomania/index.shtml and download .
Set activate keys to ctrl shift.
Try it manually first, very impressive.I tried it with html,flash and java, no problem
using autohotkey
;----------------------------------------
loop every 1 minute or whatever ?
mousemove startx,starty
send ctrl shift
mouseclickleft
mousemove endx,endy
mouseclick left
send 1 ; predined save option you set
retrieve text from clipboard and process as required.
end loop
;-----------------------------------------------------------------------
QED.
@zzaxx99 :
Go here www.structurise.com/kleptomania/index.shtml and download .
Set activate keys to ctrl shift.
Try it manually first, very impressive.I tried it with html,flash and java, no problem
using autohotkey
;----------------------------------------
loop every 1 minute or whatever ?
mousemove startx,starty
send ctrl shift
mouseclickleft
mousemove endx,endy
mouseclick left
send 1 ; predined save option you set
retrieve text from clipboard and process as required.
end loop
;-----------------------------------------------------------------------
QED.
Rebut
How can u categorically say 'NO'??? Of course there should be. I'll tell u why: I was Ctrl-Cing of selected stocks in the Mexican stock market and then pasting it with Ctrl-V in Excel (not capturing it by eye and keystroke!!!!). So if you can do that it mustn't be difficult to program it to be done every so often.
How can u categorically say 'NO'??? Of course there should be. I'll tell u why: I was Ctrl-Cing of selected stocks in the Mexican stock market and then pasting it with Ctrl-V in Excel (not capturing it by eye and keystroke!!!!). So if you can do that it mustn't be difficult to program it to be done every so often.
Rebut
How can u categorically say 'NO'??? Of course there should be. I'll tell u why: I was Ctrl-Cing selected stocks in the Mexican stock market and then pasting it with Ctrl-V in Excel (not capturing it by eye and keystroke!!!!). So if you can do that it mustn't be difficult to program it to be done every so often.
How can u categorically say 'NO'??? Of course there should be. I'll tell u why: I was Ctrl-Cing selected stocks in the Mexican stock market and then pasting it with Ctrl-V in Excel (not capturing it by eye and keystroke!!!!). So if you can do that it mustn't be difficult to program it to be done every so often.
numbertea
Well-known member
- Messages
- 257
- Likes
- 9
Easily possible with Visual Basic, c#, or C++. VB especially easy with Marshall class. You could take screen shots at whatever interval you want and then use OCR to match a font to whatever area you select on the screen. If it is financial instrument data that you are after I would recommend paying for a reliable data feed but anything is possible. Just be aware of limitations and the fact that free data is usually quite filtered. I don't know if that is what you want or not. EOD data should be easy to find for most major markets free in CSV form already. Have fun.
Cheers
