how to collect price data & perform probability analysis

If you don't know where to start, start here! Don't be afraid to ask questions.

Moderator: moderators

User avatar
bazmonaut
rank: <50 posts
rank: <50 posts
Posts: 18
Joined: Fri Jul 03, 2009 6:48 am
Reputation: 0
Location: Devon, United Kingdom
Gender: Male

how to collect price data & perform probability analysis

Postby bazmonaut » Tue Jul 14, 2009 11:24 am

(Note: moved from other post...)

This is a 'from newbie to newbie' post - I am making this up as I go along! :smt064

You will need:
- a basic understanding of Excel (google 'Excel basics') OR
- the FREE equivalent of Excel from OpenOffice (google it!)
- some data on price action in a usable format (e.g. 'CSV')

To collect data I am going to use an MT4 Expert Advisor, which will write data into a CSV file when I run 'strategy tester'. This way, I can collect OHLC for any timeframe I like, any currency pair, period etc etc.

(Anorak/geek/Germanic streak alert: unfortunately this is NOT tick-by-tick data, we are already INTERPRETING the real underlying price action into a candle OHLC format - we should still get some useful results, but just remember that we have already started down the road that leads to INDICATORS, we have already started organising the REAL data into more convenient formats. Just thought you should know that.)

I can also collect data on indicator values. The attached EA also collects data on the ADX indicator, but for now we are just interested in the price.

(Anorak/geek/Germanic streak alert: I highly recommend learning to code MetaTrader's 'MQL4' - it's not THAT difficult, and you can make a good start by reading the 'book' on www.mql4.com. The 'documentation' section is also useful.)

I'd be happy to make further modifications to this 'data collector' - just reply to this post. If you need help to change it, happy to help too. You should be able to make a reasonable go at it yourself, by looking at the code (right-click the EA in the MT4 'navigator' window and select 'Modify'), and the 'documentation' and 'book' sections of www.mql4.com).

To collect data:
- load the EA into MetaTrader (if you don't know how to do this, then search this forum)
- click the 'strategy tester' icon in the top menu bar, the strategy tester will display at the bottom of the window
- in the 'Expert Advisor' drop-down box, select the EA you just loaded
- select the symbol (we will use EURUSD in this example)
- select the period (we will use H1 in this example)
- select the model - 'tick-by-tick' takes ages to run and we're only interested in OHLC, so select 'open prices only'
- select the timeframe - we'll use the past three months (see next post for more on this)
- the EA will 'write' our data to a specific MetaTrader log file on your hard drive, to ensure there isn't any other data in there, click the 'Journal' link at the bottom of the STRATEGY TESTER window (NOT the 'terminal' window above), then right click anywhere in the 'journal' window and select 'clear all journals' (make sure you save data you have already produced otherwise you will lose it)
- click 'START', and we're off!

(You may choose to add your own commentary as the 'race' progresses, for example "...aaaaand new lower lower passes higher high on the inside OOOHHH that's a nasty new trend forming Robert whaddaya think?" etc etc)

When the test is complete, we're now ready to extract our data into Excel. To do this:
- open Excel (note: this is for Excel 2003)
- from within Excel, click 'open' (i.e. open a file)
- in the messagebox that appears, under 'files of type' (down the bottom), select 'All Files *.*'
- now open the log file, you will find this in your MetaTrader 'tester/logs' folder (e.g. C:/Program Files/MetaTrader/tester/logs/) - the file name will be in YYYYMMDD format, choose today's log file (or the day you ran the test)
- when you find the file, just click 'open' in the Excel 'open file' messagebox
- Excel MAY display a warning saying the file is locked and in use, and ask you if you want to open a 'read only' version, click 'OK' to say that you DO want to open a read only version
- Excel will display a 'Text Import Wizard' messagebox
- (step 1) under the top section 'original data type', select 'delimited', and click 'next'
- (step 2) under 'delimiters', ensure that 'comma' is checked, and ALL OTHER options are unchecked - you can now see how your data will look in Excel in the 'data preview' window in this messagebox
- click 'finish' (you don't need to complete step 3)
- you should now see your data displayed in Excel, but we're not done yet...
- now select 'Save As' from the 'File' menu
- Excel will display the 'save as' message box
- under 'file name' type a good descriptive name for your file (I will use 'data EURUSD H1 Mar09 - Jun09 v1'), make sure you remove the " quotes and the '.log' file extension
- IMPORTANT: at the bottom of the messagebox is the 'save as file type' drop-down box, select 'Microsoft Excel Workbook *.xls'
- click 'save'

IMPORTANT: all of this data comes from my retail broker, and is unlikely to match the data you get from your retail broker. Why? Because, like me, they MAKE IT UP AS THEY GO ALONG! Want to know more? Search this forum...

OK so we've run out test, collected our data... what next? Why are we doing this again? So that we can KNOW the probability of price moving in a particular direction. In this example we're going to focus on the simple things (remember, I am making this up as I go along, so it's as new to me as it is to you!). We're going to answer the question posed at the start of this post:

If price is going up NOW, how likely is it to keep going up?

I'm going to take my Excel file, and add some simple formulas to the empty columns to the right. You should be able to see the data labels easily: day, date, hour, open, high , low ,close etc etc. My Excel formula in English:
IF the colour of the current candle is green (up), then how often is the colour of the NEXT candle also green? Ditto for red candles... I have also added a 'doji' label, if open price = close price...

OK, it's not perfect, but as I said earlier, we should be able to draw some conclusions from our analysis. My formulas are in the blue cells, columns AM:AN. The summary of data is just above. Note: I have deleted some columns (volume, ADX etc) because I have been told that my 'data size is too big'...

For the RESULT please see my other post today...

bazmo
Attachments
DataCollector v1 - bazmonaut.zip
(210.75 KiB) Downloaded 138 times
DataCollector v1 - bazmonaut.zip
(210.75 KiB) Downloaded 137 times
becoming stranger in a strange land

Please add www.kreslik.com to your ad blocker white list.
Thank you for your support.

User avatar
monolisa
rank: 150+ posts
rank: 150+ posts
Posts: 351
Joined: Fri Jun 12, 2009 11:38 am
Reputation: 0
Location: All Quiet on the Western Front
Gender: None specified

Postby monolisa » Tue Jul 14, 2009 12:13 pm

Hi bazmo,

Thanks for the EA. I wonder if the data could be simply exported from the History Centre (although it doesn't allow custom period export).

Lis
"Know your enemy and know yourself, find naught in fear for 100 battles. Know yourself but not your enemy, find level of loss and victory. Know neither your enemy or yourself, wallow in defeat every time." - Sun Tzu

User avatar
bazmonaut
rank: <50 posts
rank: <50 posts
Posts: 18
Joined: Fri Jul 03, 2009 6:48 am
Reputation: 0
Location: Devon, United Kingdom
Gender: Male

Postby bazmonaut » Wed Jul 15, 2009 7:38 am

Hi Lis

Yes I think it can be exported from the history center, but by using an EA I can collect data on indicators, and do calcs within the EA without having to calculate them in Excel... also thought this would be helpful for less tech-minded people.

Where are you in Australia? I'm from Melbourne, been in the UK for 8 years but still call Australia home...

bazmo
becoming stranger in a strange land

Please add www.kreslik.com to your ad blocker white list.
Thank you for your support.


Return to “beginners forum”