The tick data section of eareview.net is a detailed guide that will lead you through the whole process of tick data backtesting, starting from where to acquire free historical Forex tick data, how to download it and how to use it in backtesting Metatrader 4 expert advisors to obtain a 99% modeling quality. If you’re not sure what backtesting is, it’s probably a good idea to purchase the Metatrader Backtesting and Optimization Course, which is geared toward people that are new to Forex and Metatrader 4 backtesting.

This page is divided into several sections:

Scope

In general, backtesting using the data from the MT4 history center might be good enough for EAs that are not scalping or pip hunting. However, if you’re dealing with such an EA or any kind of EA that closes trades within 1-15 pips, even the smallest feed differences might have a very large impact.

The issue is caused by the Metatrader terminal not having access to the real tick data, but only to minute bar data in the best case, which forces it to give your strategy backtest “false” ticks generated through a process of interpolation using the data for the smallest timeframe available. This is most likely not important to an expert advisor that uses stoploss and takeprofit targets of over 100 pips, but in the case of robots that attempt to scalp a few pips here and there, your backtest could be completely misleading.

So, it is very important to try testing using data having a quality that is as high as possible which is why I put together some resources, all of which I use in my backtesting when needed.

Tick data guides

  1. How to download free tick data – details the download process using several free tick data sources: Dukascopy, Oanda, Pepperstone, Integral, MB Trading and Gain Capital.
  2. How to prepare your tick data for Metatrader 4 – guide to converting the tick data to a format compatible with Metatrader 4 (from CSV to FXT).
  3. How to backtest using tick data with Metatrader 4 – a review of the options available for using tick data with the Metatrader 4 platform.
  4. FAQ & Troubleshooting
  5. Downloads

The Walk Forward Analyzer

Not directly related to tick data but with built-in support for it, the Walk Forward Analyzer is an excellent tool that lets you optimize your Metatrader 4 expert advisors in steps, in a technique called Walk Forward Analysis. Simply put, you optimize your EA for say 3 months, then you test it for the next 1 month to see if the best parameters resulting from the optimization work fine on out-of-sample data, then you optimize it further on the next 3 months and so forth. This tool lets you automate the whole process and does all the runs for you, providing an exhaustive set of configuration parameters and a neat optimization report at the end. The Walk Forward Analyzer is a must-have for anyone doing serious EA development. But don’t take my word for it, visit the website and download your copy – the WFA used to be priced around $30 but recently the author decided to provide it for free.

History

Since there are frequent updates to the tick data tools, I decided to keep a Tick data changelog that lists all the changes that the scripts have gone through.

Also, for those interested, the old tick data page is still available but a lot of the information there is now obsolete.

  • #1 written by Hyperdimension October 14, 2009 (7 years ago)

    birt, I see that you’ve now taken your MetaTrader patch public with this web site.

    I think your patch is one of the greatest recent achievements in the field of MetaTrader trading system development, backtesting and optimization. It’s a big win for traders in the war against retail brokers who want to take our money. MetaQuotes themselves are on the side of the brokers because they’re their revenue source. So it’s no wonder that MetaQuotes changed their software to prevent backtesting over tick data. Brokers don’t want traders to optimize EAs for their particular price feeds, otherwise either traders may discover profitable anomalies, or they may find that their EA doesn’t actually work on real data, only on smooth indicative data like MetaQuotes’ data that you can download through the MetaTrader terminal History Center. (I call such data “junk” data).

    I’ve mentioned many times that backtesting should be done on an offline terminal. The terminal should have once been online (so that internal variables have values set) but has now been taken offline. It just makes scientific sense to not have a platform or foundation whose internal variables are continuously changing whilst you conduct tests on it. You also need to remember that the LiveUpdate process may start one day if you keep the terminal connected, and this may render the patch useless due to changes in terminal.exe.

    To take a terminal offline you can simply delete the account from the MetaTrader Navigator pane on the left hand side of the window.

  • #2 written by birt October 16, 2009 (7 years ago)

    Friend of mine points out that Dukascopy prohibits data redistribution, so I had to remove the torrent links and wrote a windows how-to instead.

  • #3 written by birt October 19, 2009 (7 years ago)

    Updated the FXT creation scripts to fix a problem with negative or zero volumes in the source data.

    As a side note, MT4 generates ticks based on the history volume, which leads to extremely slow FXT interpolation for new timeframes if using the history generated by the script. Thus, the new version features a parameter to control whether the real volume is used, which is disabled by default. Enable it only if you actually need & use the volume data.

  • #4 written by Futron November 6, 2009 (7 years ago)

    Birt,

    Great BLOG !!

    One problem I Picked up. When running the Dukascopy script it does not generate the .hst files. I get the message that the generation has been completed but in experts tab it says only 1 bars has been finalized in header and all the .hst files are 1KB in size. This is generating data for 200901 – 200910

    Hope you can help me out here.

  • #5 written by Taz November 7, 2009 (7 years ago)

    What in the download.php script limits the downloading of data to only go back to early 2007? I see that dukascopy has data much farther back for most pairs. I viewed the php file but I don’t see where it says to start in 07.
    Thanks for any help, I am wanting more backdata for some pairs.
    Great site and great info, thanks for all the work involved!

  • #6 written by Futron November 8, 2009 (7 years ago)

    Birt

    i think my problem comes in with the process_duskacopy script.

    I checked the .csv file created and there were no dates and times although the fraction tick seconds is there.

  • #7 written by Futron November 8, 2009 (7 years ago)

    Found the problem.

    change this line in process_dukascopy_data.php
    fwrite($outfd, gmstrftime(“%Y.%m.%d %T”, $timesec).”$timems,$bid,$ask,”.number_format($bidvol,0,”,”).”,”.number_format($askvol,0,”,”).”\n”);

    to

    fwrite($outfd, gmstrftime(“%Y.%m.%d %H:%M:%S”, $timesec).”.$timems,$bid,$ask,”.number_format($bidvol,0,”,”).”,”.number_format($askvol,0,”,”).”\n”);

  • #8 written by birt November 9, 2009 (7 years ago)

    Futron: good find. I suppose it was caused by the regional settings on your computer, but the solution you implemented is more generic. Thanks for posting it here. I will implement it in the scripts next week.

    Taz: the array at the very beginning of the file is the starting date for each currency in unix timestamp format (seconds since 01 january 1970). Look into mktime() (see the PHP documentation) to generate such values for different dates. However, while Dukascopy does have data older than that, I’m afraid it’s not in tick format thus not very helpful.

  • #9 written by Taz November 9, 2009 (7 years ago)

    Ok, thank you Birt for the answer. After grabbing all this and doing some conversions, the 2g limit in mt4 kind of prevents u from really using more than that anyhow, so I’ll just stick with the 3 years. Appreciate all you have shared here!

  • #10 written by Michael Wales November 13, 2009 (7 years ago)

    Hi,

    Downloaded and ran your downlad php script and got the following error message. I would really appreciate it if you can help me out.

    WARNING: did not download http://www.dukascopy.com/datafeed/AUDJPY/2007/03/04/05h_ticks.bin (1175662
    800) – error code was 502
    Content was:

    Bad Gateway!

    */–>

    Bad Gateway!

    The proxy server received an invalid
    response from an upstream server.

    The proxy server could not handle the request GET /datafeed/AUDJPY/2007/03/04/05h_ticks.bin.
    Reason: Error reading from remote server

    If you think this is a server error, please contact
    the webmaster.

    Error 502

    http://www.dukascopy.com

    Fri Nov 13 09:32:59 2009
    Apache/2.2.4 (Linux/SUSE)

    Thanks in advance
    Michael Wales

  • #11 written by birt November 15, 2009 (7 years ago)

    Hi,

    Quoting from the windows how-to above:

    If you get any strange errors, run the process again when it’s finished – it will only download any files that were missed in the first step due to network errors.

    The errors you got are exactly the errors I’m talking about. Dukascopy might be using a proxy server to do some kind of load balancing and in some rare cases it simply fails. Running the script once more will typically download these files properly.

  • #12 written by birt November 15, 2009 (7 years ago)

    Added Futron’s fix to the scripts archive. Also added the date to the archive name to avoid potential version confusion. Thanks again for posting your solution, Futron.

  • #13 written by birt November 23, 2009 (7 years ago)

    While talking to a friend about the unix timestamps and regular dates, I stumbled upon http://www.epochconverter.com/ and thought it’s worth posting here. It’s very useful for converting from a regular date to a unix timestamp and the other way around, which comes in pretty handy when fiddling with the download script. I also edited the article to mention it.

  • #14 written by birt November 24, 2009 (7 years ago)

    One of the things that always annoyed me was the fact that you can open a single terminal instance for each MT4 installation. If you want to take advantage of modern CPUs and perform several backtests at once, you have to manage windows junction links with several directories common between installations and overly complicated stuff like that.

    So, I decided to adjust the FXT loader to also include the capability to run the terminal client multiple times from the same installation directory. I don’t know why I didn’t do this earlier, it sure would’ve helped me a lot. Download it at http://eareview.net/download/fxt_multiloader.zip

    For those interested, the patched offsets are:
    0x4BBDFE from 3B to 00
    0x4BBDFF from 04 to 00

    • #15 written by alex May 26, 2011 (5 years ago)

      Birt,

      So what does the fxt loader do? Does it just start MT4 instances which could read the same FXT file in parallel?
      It does not seem to work on build 229.

      Kind Regards,
      Alex

      • #16 written by birt May 27, 2011 (5 years ago)

        As of now, the loader only lets you start multiple instances of MT4 from the same directory, thus making it possible to run multiple concurrent backtests on the same FXT.

        Re: build 229, did you use the specific 229 loader from the loader archive posted in the article (the one with a bunch of loaders inside)?

  • #17 written by Owen November 26, 2009 (7 years ago)

    Firstly I want to thank Birt for this great resource!

    Secondly I am getting an error when I get to the extract to CSV step – I am Typing php process_dukascopy_data.php AUDJPY 200702 200911 AUDJPY.csv and it is spitting out a multitude of errors saying:
    Warning: Invalid CRT parameters detected in C:\php\process_dukascopy_data.php on line 102
    PHP Warning: Invalid CRT parameters detected in C:\php\process_dukascopy_data.php on line 102

    Is this normal?

  • #18 written by birt November 26, 2009 (7 years ago)

    No, that’s not normal at all. It looks like there’s a problem with your VC++ runtime time format strings. Please copy/paste the following into a .php file, try to run it and let me know what errors it outputs:

    < ? $t = time(); $a = gmstrftime("%Y.%m.%d %H:%M:%S", $t) $a = gmstrftime("%Y", $t); $a = gmstrftime("%m", $t); $a = gmstrftime("%d", $t); $a = gmstrftime("%H", $t); $a = gmstrftime("%M", $t); $a = gmstrftime("%S", $t); ?>

  • #19 written by Nicola November 30, 2009 (7 years ago)

    Hi Birt,
    many thanks for this great resource! Everything is accurate and clear. Best,

    Nicola

  • #20 written by Owen November 30, 2009 (7 years ago)

    Hi Birt, Thanks for the reply

    Ran the script and didnt get any errors, but I think you may have left out the <?php

    When I included that I got an error saying
    PHP Parse error: parse error in \sample.php on line 4

    Any Ideas?

  • #21 written by birt November 30, 2009 (7 years ago)

    Yeah, you’re right, I forgot the

    • #22 written by RJ1 December 15, 2009 (7 years ago)

      Hi Birt,

      I had the same parse error in sample.php. I also modified according to Futron, but that also contains errors. Please forward me the corrected scripts if you have one. TQ.

  • #23 written by RJ1 December 15, 2009 (7 years ago)

    I tried the process_dukascopy_data.php included, but I got error messages (a lot) about “Warning: Invalid CRT parameters detected in D:\Php\process_dukascopy_data.php on line 102”

    Please help..

    -RJ1-

  • #24 written by birt December 15, 2009 (7 years ago)

    Unfortunately, it looks like it’s a problem with your VC9 runtime. There seems to be a problem with some CRTs that have an incomplete range of time format strings. You can try installing the redistributable at http://www.microsoft.com/downloads/details.aspx?familyid=A5C84275-3B97-4AB7-A40D-3802B2AF5FC2&displaylang=en – if that doesn’t help, I’m afraid I don’t know what to advise other than running it on a different system.

  • #25 written by gigi December 19, 2009 (7 years ago)

    Somebody mentioned that he got banned when downloading with a script Dukas tick data. Did this happen to you?

    I’m thinking of writing a Python script to download it.

    I already have scripts which download FXCM tick data (no manual steps required) using their Order2Go API. It takes around 1 day to download 1 year of tick data (their servers are throttled).

    I also have scripts which download/sort/de-duplicate/reindex Gain Capital ticks.

    But I would like the Dukas volume data.

  • #26 written by birt December 19, 2009 (7 years ago)

    It’s the first time I hear about someone getting banned for that.

    I’m not sure I understand why you want to write a Python script to download the tick data instead of using the readily available PHP one?

  • #27 written by gigi December 19, 2009 (7 years ago)

    I have a custom compressed tick storage format, so I would need to write a script to convert the data anyway.

    It seems I misunderstood, the person was banned for another data set. Here’s the banning I was referring to, and you were part of the discussion.

    Anyway, it’s good to know I can download in peace 🙂

    —–

    birt wrote:
    I’ve been doing some googling and it turns out Dukascopy has free tick data.

    I check the dukascopy site and found the page you were referring to. I’m not sure why I didn’t know of it before; either it’s a recent addition or my attention was focused on the CSV data export page, which is a different data set – indicative data from a number of different sources, which we don’t want; we want data from a single trade server. I had a lot of problems downloading that data too, my IP address was banned and using anonymous proxies worked only for a while.

  • #28 written by Sciocchino December 30, 2009 (7 years ago)

    I think there is an error in the process_dukascopy_data.
    Where you “print” the milliseconds (the $timems variable, line 82 and 104), you don’t pre-pad them. An example:
    You write 16:00:00.001 as 16:00:00.1
    This COULD create a problem because ticks COULD be in the wrong order (16:00:00.1 shoud be interpreted as 16:00:00.100).

  • #29 written by birt December 30, 2009 (7 years ago)

    Although this does not constitute a problem (the ticks are supposedly correctly ordered to begin with), I fixed it and updated the download links. The only potential problem would’ve appeared if you sorted the CSV files with another tool before making FXT files from them, although I don’t think there’s any reason to do that.

    Any 16:00:00.1 will now show as 16:00:00.001. Naturally, as a side effect, the resulting CSV files will be slightly larger.

    Thanks for bringing it to my attention.

  • #30 written by Susan January 23, 2010 (7 years ago)

    Great work Birt. Before i saw your solution i downloaded an older version of MT4 (Build 208) which has the Recalculate functionality that also allows backtesting with Tickdata. (That functionality was removed since build 210)

    (I prefer to build tickhistory from live tickdata from my own broker Thecollectivefx, i do not have a trading account with Dukascopy)

    I am a bit confused what to use, your solution or just build 208.

    Maybe you or someone can advise what the most important advatages/disadvantage will be when we compare both ?

    Thanks in advance.

  • #31 written by birt January 24, 2010 (7 years ago)

    Build 208 works, too, but you won’t be able to easily backtest EAs using some of the more recent MQL changes.

    As for a Dukascopy account, you don’t need one to download the tick history data.

    • #32 written by Ludo March 9, 2011 (5 years ago)

      VERY GOOD WEBSITE AND MANY THANKS!!
      One question: the nice thing about using build 208 was that it inhibited fractal interpolation if you let the Recalculate window unchecked – i.e., the behaviour of the MT terminal to recreate a novel fxt file with fictional interpolated values from the M1 values.
      With your nice patches from what I understand you also bypass fractal interpolation in MT4 and have MT4 read the real tick data generated by the Dukascopy source. In addition, your patches work with newer versions of MT4, therefore allowing you to test newer EA’s that may leverage the features embedded in newer versions of MT4. Can you confirm that?

      • #33 written by birt March 9, 2011 (5 years ago)

        Yes, you got it right, that’s exactly what the patch script does.

  • #34 written by Susan January 24, 2010 (7 years ago)

    Ok thanks, i’ll try to learn and have a last question.

    I understand that your solution allows to set the spread value at any fixed value we like to backtest with.

    I am not sure but i thought that this was not possible when using build 208, because i read that it uses the fluctuating spread during backtest with the values as reflected in the tickdata history.

    Is it possible to use your solution also with the fluctuating spread values as reflected in the tickdata history ?

    Thanks in advance.

  • #35 written by birt January 25, 2010 (7 years ago)

    Unfortunately, none of the MT4 builds is able to use variable spreads in backtesting.

    I’ve “met” the Metatrader client shortly after MT4 was released at which time the spread in backtesting was fixed. Using the most recent build, it is still impossible to use variable spread in backtesting and the FXT file structure remains almost the same as it was when MT4 was released. Although I didn’t check out build 208 and a few others in between, it’s safe to say that whoever wrote that build 208 is using fluctuating spread is mistaken.

  • #36 written by Susan January 25, 2010 (7 years ago)

    Thanks for your info, i concluded to setup your solution instead of using build 208.

    I would like to optimize my EA once a week in the future. But if we run more than one EA this can be a boring and timeconsuming work. Because your tickdata backtest solution is for a lot people very useful, maybe a “second” step for your informative website is to work on another revolutionair project, to automate the optimization process. On the mql4 website i found an article translated from Russian. http://articles.mql4.com/506 Currently it’s just a thought but it is surely something that has my interest to test in the future.

  • #37 written by birt January 26, 2010 (7 years ago)

    There is already such an auto optimizer available. As far as I know it’s made by a Russian and costs around $40.

  • #38 written by Joe February 3, 2010 (6 years ago)

    I am use this process to download data for running backtest using Forex.com platform.

    Everything is functional except my model quality remains at 25%…

    Are there setting that I am missing in this process…?

    • #39 written by birt February 3, 2010 (6 years ago)

      It sounds like the FXT file wasn’t present at the correct location or you ran the MT4 client without the loader… Did you copy it? Is it still there? What size does it have?

      • #40 written by Joe February 3, 2010 (6 years ago)

        Ok I missed a process step… I am running using the loader all files are in there locations… The problem I am having is the backtest is ending before the end date… The journal is not reporting any errors… There is sufficient capital so I dont see any issues… Can you help…? The FXT file is 2.4 GB for the EU 60min and yes the file is still there…

        • #41 written by birt February 3, 2010 (6 years ago)

          As I mentioned in my article, MT4 is unable to read files past 2 GB. The time interval you selected is too large.

          On a different note, I find it strange that you were able to produce a 2.4 GB FXT, normally the script should just stop at 2 GB.

          • #42 written by Joe February 3, 2010 (6 years ago)

            Ok thanks… I will go back and recreate the data on a lower TF… Thanks for pointing this out…

  • #43 written by Joe February 3, 2010 (6 years ago)

    I ran the backtest without the FXT file. I used only the 1min HST file… Bad Idea…?

    • #44 written by birt February 3, 2010 (6 years ago)

      I wrote a very detailed description of the whole process. Please read it and apply it fully. If you ran the test using M1 HST data, it means you simply did not read everything and I don’t even want to try guessing which steps you skipped. To sum it up, RTFM.

  • #45 written by birt February 4, 2010 (6 years ago)

    Apparently we reached the max reply depth so I’m posting a new comment entry: the chart timeframe has absolutely nothing to do with the size of the resulting file. By “time interval” I mean the difference between the start date and end date, which you can configure in the script parameters.

    • #46 written by Joe February 4, 2010 (6 years ago)

      Thanks again… Let me review that information as well…

      • #47 written by Joe February 4, 2010 (6 years ago)

        Success…! Thanks for your patience and assistance..This is great process…

  • #48 written by Susan February 5, 2010 (6 years ago)

    Birt, i could not find the russian comercial version you mentioned but instead i found this one: http://blog.iticsoftware.com/

    I would like to combine it with your above solution to have auto-optimizing with 99% quality. Especially for auto-optimizing of EA’s that trade on low TF like M1 or M5. But do you think that such combination is able to work live?

    • #49 written by birt February 5, 2010 (6 years ago)

      Well, I have no idea about that tool, but you can surely schedule some scripts to update & replace the FXT files. So if you can somehow get that tool to start MT4 via the loader, it should work. If you can’t get it to start a different executable file, you can either patch terminal.exe manually (the offsets are above) or you can keep a copy running with the loader, in which case running terminal.exe again will just bring that window to the foreground.

  • #50 written by Susan February 5, 2010 (6 years ago)

    Thanks for your vision.
    Although i have no programming experience, and so it could become difficult for me to setup scripts to update and replace the FXT files, i’ll try to start with it and just see how far i can come. It could become a powerful combination if it work together.

  • #51 written by Steve C February 9, 2010 (6 years ago)

    Hi this well documented and a great blog.

    I have a question regarding the following
    “Select the timeframe that you wish to generate the FXT for (if you want to backtest on H1, then select H1 as the chart timeframe). •Double click the Dukascopy2FXT script in the navigation panel (it’s in the scripts section) and configure the parameters in the window that pops up.”

    Do you have to do this for each timeframe you want to use, for exaple the M5, M15 etc?

    • #52 written by birt February 9, 2010 (6 years ago)

      Yes, you must generate a new FXT for each timeframe you want to use.

  • #53 written by Miro February 25, 2010 (6 years ago)

    The http://www.epochconverter.com/ seems to be offline (or maybe only for me? 🙂 )
    So I am using
    http://www.onlineconversion.com/unix_time.htm

    For sure there is many many other sources…
    It is just my two (one) cents.

    Regards,
    Miro

  • #54 written by Juliano CS February 25, 2010 (6 years ago)

    Hi Birt, first thanks a lot for your work and your time.. I really appreciate it.

    But i’m having problems here, i make all steps but when i open a offline chart EURUSD30M to start the backtest, the history are totally wrong…

    First -> downloaded all ticks with your script from Dukascopy.
    Second -> converted EURUSD 2009 01 -> 20010 01
    CSV file period to FXT with script with Create HST True and HST STARTDATE 2009.01.01 – ENDDATE 2009.12.31.
    Third -> put the file in the correct folders.
    Fourth -> open MT4 with loader and tried to open EURUSD 30M generated
    And shows me this
    http://img682.imageshack.us/img682/4795/dukaserror.jpg

    What is wrong?
    Regards

    Juliano

    • #55 written by birt February 26, 2010 (6 years ago)

      You don’t have to open an offline chart. Just start the backtest as you normally would.

  • #56 written by O.Z. February 26, 2010 (6 years ago)

    Hello,
    first of all I would like to thank you for your work. It surely helps a lot of people.

    However I ran into a litte problem with which I hope you can help me:
    I downloaded all the data and whe I try to run “process.bat” I get this error:
    “Fatal error: Class ´ZipArchive´ not found in C:\php\process_dukascopy_data.php on line 59”
    I would really apreciate if you could help me out. Thank you,
    Ondrej.

    • #57 written by birt February 26, 2010 (6 years ago)

      This is a problem that depends on your PHP version and on the OS you run it on. My windows with php 5.3.0 didn’t have an issue with it. If you get the windows 5.3.x version, it should work out of the box. If you want to get it to work with an older version, take a look at http://www.php.net/manual/en/zip.setup.php

      • #58 written by O.Z. February 26, 2010 (6 years ago)

        Hi,
        tried to download newer php and it works fine. Just had to change directory in “process.*” aswell.

        Thank you again for your help,
        Have a nice weekend,
        Ondrej.

      • #59 written by Csoky March 21, 2010 (6 years ago)

        Dear Birt!

        Thanks for this great site!
        I have problem with ZipArchive too.
        My computer is mac with OSX (Snow Leopard).
        PHP 5.3.0 run on it.
        I got the same error message like above.

        “Warning: 0 sized EURUSD/2007/03/01/20h_ticks.bin
        EURUSD/2007/03/01/21h_ticks.bin
        PHP Fatal error: Class ‘ZipArchive’ not found in /Users/csoky/Documents/DUKASCOPY/dukascopy_php_scripts/process_dukascopy_data.php on line 59”

        Do you have an idea about this problem?

        Thanks a lot!

        • #60 written by birt March 22, 2010 (6 years ago)

          Edit your PHP ini file and enable the ZIP extension. If you’re not sure how to do it, a quick google search should tell you about it.

  • #61 written by danih March 11, 2010 (6 years ago)

    Hello!

    Thanks for the great tools.
    My question: is there a workaround regarding GMT offsetting? For example, I have an alpari.uk acc, where broker time is GMT+1, so H4 candles are different than with a GMT broker. How to offset that easily?
    Any way to take it into account while converting?

    • #62 written by birt March 11, 2010 (6 years ago)

      You can easily do one of the following:

      1. Edit Dukascopy2FXT.mq4 and inside the ReadNextTick() function you can add “cur_time += 3600 * GMT_Offset;” right after “cur_time=StrToTime(date_time);”

      2. Edit process_dukascopy_data.php and in the decode_ducascopy_bin(…) function, right before “$timesec =…” you can do “$time = bcadd($time, 1000 * 3600 * $gmtoffset)”.

      There are also other ways to do it.

  • #63 written by danih March 13, 2010 (6 years ago)

    Thanks birt, I’ll try those and see!

  • #64 written by danih March 13, 2010 (6 years ago)

    I am trying the 2nd option, editing the php file.
    Inserting
    $time = bcadd($time, 1000 * 3600 * $gmtoffset);
    before the “$timesec =…” line gives an error, I believe gmtoffset is not defined.
    However, when removing gmtoffset and inserting this line:
    $time = bcadd($time, 1000 * 3600);
    it seems to work.

    • #65 written by birt March 13, 2010 (6 years ago)

      Well, naturally, you were supposed to add a $gmtoffset variable and set it to 1 since your GMT offset was 1. Sorry, I thought it was obvious so I didn’t mention it. However, removing it is the same as multiplying by 1 so it will work as well in your case.

  • #66 written by greg March 15, 2010 (6 years ago)

    Hi, I am very new to using MT4 and I wanted some help regarding the downloaded files. For example, I am in the process of downloading the different currencies by using the above notes, but I noticed that the files that have been downloaded are .bin format. How can I get the CSV file in order to import it into MetaTrader4 as expected. Or am I missing something?

    Thanks

    G

    • #67 written by birt March 15, 2010 (6 years ago)

      You must’ve missed reading the howto, because it explains what you have to do to get the data in CSV format. There’s a PHP script whose sole purpose is doing just that.

      • #68 written by greg March 15, 2010 (6 years ago)

        Hi Birt,

        Thanks for the response, I have now downloaded the data and have managed to import it into the Metatrader platform. I have only imported for 2007on AUDUSD. How can I get the chart on MT4 to change and show the new data and not show the older data that was there before?

        Thanks

        G

        • #69 written by birt March 15, 2010 (6 years ago)

          The data is not supposed to show on the chart just like that, although you can use open offline on the FXT, I guess. The whole idea is to use it in backtests, not to inspect it on the chart. If you’re new to the MT4 platform, I suggest avoiding to use tick data unless you need to test a scalper and you know it could make a difference.

  • #70 written by jmwilk March 29, 2010 (6 years ago)

    Hi Birt,

    When I try to download the FXTLoader software from your links, my virus scanning is picking it up as a trojan… Can you take a look…?

    Thanks,

    JMWilk

    • #71 written by birt March 29, 2010 (6 years ago)

      It’s most likely something wrong with your antivirus. Anyway, if you don’t trust it, don’t use it, it’s as simple as that. I provided the offsets if you’d like to grab a hex editor and patch it manually, in which case you won’t need to use the loader.

  • #72 written by Andy March 31, 2010 (6 years ago)

    Hello Birt,

    Thank you so very much for this info. It worked without a hitch and your directions were very clear and easy to follow. Much gratitude.

    Regarding GMT offset. I just downloaded the beta Dukascopy MT4 terminal and it appears to be GMT +1. Does this effect the assumption the tick data should be tested with GMT 0?

    • #73 written by birt April 1, 2010 (6 years ago)

      Good to hear it worked. The Dukascopy MT4 terminal being GMT+1 has nothing to do with the data – the data is still GMT 0. Their MT4 terminal is not directly implemented by them, it’s done in a partnership with FXOpen and I assume they control the GMT offset for the MT4 terminal.

  • #74 written by Ian April 3, 2010 (6 years ago)

    Hi Brit could you explain why it is neccessary to convert csv file to hst files ? I was looking at the import option in mt4 history center it seems it can import csv files.
    Also the import function has a shift box i assume this is a time shift for offsetting the time in case the data was generated in a different time zone ?

    • #75 written by birt April 7, 2010 (6 years ago)

      It’s necessary to convert the CSV files to HST because sometimes the EAs use indicators that operate on timeframes different than the one you’re using for the backtest, in which case the history files are used. If you don’t generate HST files, the history data will not match the FXT tick data.

      As for the import function, I strongly doubt it works with the CSV files generated from the Dukascopy data. The shift box is most likely for the GMT offset (I haven’t tested it), but if you want to use the tick data with a GMT shift you shouldn’t be looking at that. Check out the comments above, you will find a solution for it among them.

  • #76 written by DhaoS April 11, 2010 (6 years ago)

    Hi,

    I took the data from dukascopy but they have a lot of 0 sized files…i tryed to go directly with my navigator to get correct file but it isn’t a bug in your script : they do have lots of 0 size files

    Also i managed to get the data from a forum and when coverted to fxt mt4 just don’t recognize it and i can’t even open the tick data file

    The patch for build 226 doesn’t work either for me, it just rebuild the data whatever i already have a fxt or not. I had to take the old one with the recalculate box

    That’s very kind to have put all your tools, a so nice article and a so good rating but for me nothing actually works. I guess i’ll try the gain capital ticks

    Anyway i like your site, some reviews are hilarious 🙂

    • #77 written by birt April 12, 2010 (6 years ago)

      Most of the 0 sized files are during the weekends. The Dukascopy data is very good, although there are 4-5 missing files per currency, probably due to daylight savings.

      As for nothing working, you’re probably doing something wrong. With the info you provided, it’s pretty much impossible to figure out what, though. Also, I just rechecked the .EXE loader for build 226, it works fine. Perhaps you’re trying to use it on build 225? Inspect the about box.

      Other than that, thanks for the kind words.

  • #78 written by poolpips April 11, 2010 (6 years ago)

    Hi, birt
    Thank you so much for good information here.
    I would like to download 10sec data as well because I can get longer history data (from 2003 july~) than tick data.
    Can you change your script to download 10sec data from Dukascopy?

    • #79 written by birt April 12, 2010 (6 years ago)

      I’m afraid I don’t have the time for that, sorry. It’s not really a trivial change.

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <s> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top