The tick data section of eareview.net is a detailed guide that will lead you through the whole process of tick data backtesting, starting from where to acquire free historical Forex tick data, how to download it and how to use it in backtesting Metatrader 4 expert advisors to obtain a 99% modeling quality. If you’re not sure what backtesting is, it’s probably a good idea to purchase the Metatrader Backtesting and Optimization Course, which is geared toward people that are new to Forex and Metatrader 4 backtesting.

This page is divided into several sections:

Scope

In general, backtesting using the data from the MT4 history center might be good enough for EAs that are not scalping or pip hunting. However, if you’re dealing with such an EA or any kind of EA that closes trades within 1-15 pips, even the smallest feed differences might have a very large impact.

The issue is caused by the Metatrader terminal not having access to the real tick data, but only to minute bar data in the best case, which forces it to give your strategy backtest “false” ticks generated through a process of interpolation using the data for the smallest timeframe available. This is most likely not important to an expert advisor that uses stoploss and takeprofit targets of over 100 pips, but in the case of robots that attempt to scalp a few pips here and there, your backtest could be completely misleading.

So, it is very important to try testing using data having a quality that is as high as possible which is why I put together some resources, all of which I use in my backtesting when needed.

Tick data guides

  1. How to download free tick data – details the download process using several free tick data sources: Dukascopy, Oanda, Pepperstone, Integral, MB Trading and Gain Capital.
  2. How to prepare your tick data for Metatrader 4 – guide to converting the tick data to a format compatible with Metatrader 4 (from CSV to FXT).
  3. How to backtest using tick data with Metatrader 4 – a review of the options available for using tick data with the Metatrader 4 platform.
  4. FAQ & Troubleshooting
  5. Downloads

The Walk Forward Analyzer

Not directly related to tick data but with built-in support for it, the Walk Forward Analyzer is an excellent tool that lets you optimize your Metatrader 4 expert advisors in steps, in a technique called Walk Forward Analysis which is also described in the Metatrader Backtesting and Optimization Course. Simply put, you optimize your EA for say 3 months, then you test it for the next 1 month to see if the best parameters resulting from the optimization work fine on out-of-sample data, then you optimize it further on the next 3 months and so forth. This tool lets you automate the whole process and does all the runs for you, providing an exhaustive set of configuration parameters and a neat optimization report at the end. Priced at $29.95, it’s a veritable steal; the Walk Forward Analyzer is definitely a must-have for anyone doing serious EA development. But don’t take my word for it, visit the website and download the 14 day trial version. To top it all, Andrew – the programmer – is also very prompt with tech support as well as feature requests that make sense.

History

Since there are frequent updates to the tick data tools, I decided to keep a Tick data changelog that lists all the changes that the scripts have gone through.

Also, for those interested, the old tick data page is still available but a lot of the information there is now obsolete.

  • #1791 written by fxtech January 15, 2012 (2 years ago)

    Is any one else having trouble logging in to jforex besides myself and Eddie? I just opened a new demo account this weekend, but it says “authentication failed”. Could it be because it is the weekend?

    • #1792 written by birt January 15, 2012 (2 years ago)

      Dukascopy might be having some issues, the tick data files are all 404 as well. Perhaps a server migration of some sort.

  • #1793 written by Mat January 15, 2012 (2 years ago)

    Same problem here for jforex.
    However running ducascopier without the convert in csv option is very very fast, the there will be the new jforex2fxt script it will be working all perfectly…..

  • #1794 written by Mat January 15, 2012 (2 years ago)

    By the way, now I am processing the euraud data with php to create the csv file.
    Sometimes I get some warning about. Issuing data file in some folder…. It is not a Saturday or a Sunday …. So are those error in the tick data? Is this normal? Or should I use the php script to grab them course is never failing?

    • #1795 written by birt January 15, 2012 (2 years ago)

      It’s probably nothing you should be worried about but I can’t tell you anything else without the exact error messages.

    • #1796 written by birt January 15, 2012 (2 years ago)

      Also, if you continue posting comments here please either correct your email address or otherwise stop clicking the “notify me” box when using an invalid email address.

  • #1797 written by trader January 16, 2012 (2 years ago)

    May i know whether i can optimize my ea using the tick data?

    • #1798 written by birt January 16, 2012 (2 years ago)

      The patch script might work with optimization with builds 402 or below. Mind you, there’s no guarantee it will and you might run into other problems such as the EAs not working with those builds.

      This month I will release a commercial tool that will allow optimization with tick data with all builds.

      • #1799 written by Alex January 22, 2012 (2 years ago)

        Hi Birt, will the commercial tool also enable us to use the Work Forward analyser (disabling overwriting permanently, so the script doesn’t need to run at the beginning – I guess this is causing the incompatibility at the moment). Thanks

      • #1800 written by Alex January 22, 2012 (2 years ago)

        Ah and also, will we be able to use multiple instances with the Walk Forward Analyzer (which starts up MT4 via command line)? Multiple instances really speed up my testing..

        • #1801 written by birt January 22, 2012 (2 years ago)

          The current beta already works with the Walk Forward Analyzer. The existing files don’t get overwritten, but you have to make sure that your FXTs have the whole range required by the WFA.

          As for starting multiple instances, I don’t know. If the WFA supports that, then yes, definitely. Running the loader repeatedly allows running multiple instances from the same folder.

  • #1802 written by trader January 16, 2012 (2 years ago)

    My metatrader always restarted everytime i press the start button on strategy tester. So i can’t backtest. Do you have the solution?

    • #1803 written by pipsaw January 16, 2012 (2 years ago)

      I experienced the same issue couple of times. Try applying the Birt’s patch as first thing after starting MT4 terminal. That worked for me!

  • #1804 written by Tyler January 16, 2012 (2 years ago)

    Dukascopier is still reporting nothing but 404 errors trying to find data on Dukascopy’s website. Exmaple:

    D:\Forex\DukasCopier\EURUSD\201201\21h_ticks.bin – The remote server returned an error: (404) Not Found.
    D:\Forex\DukasCopier\EURUSD\201201\22h_ticks.bin – The remote server returned an error: (404) Not Found.

    Is anyone else having any luck?

    • #1805 written by pipsaw January 16, 2012 (2 years ago)

      Sorry, forgot to hit the reply button earlier –

      Tyler I dont think the issue is with Dukascopier, no luck on php or Dukascopy website either. Interestingly, the data is gone from the start of 2008. This is what I have from php script (i can’t get anything after this timestamp) –

      Info: Downloading EURUSD starting with
      WARNING: missing file http://www.dukascopy.com/datafeed/EURUSD/2008/00/01/00h_ti
      cks.bin (1199145600)

  • #1806 written by pipsaw January 16, 2012 (2 years ago)

    Tyler I dont think the issue is with Dukascopier, no luck on php or Dukascopy website. Interestingly, the data is gone from the start of 2008. This is what I have from php script (i can’t get anything after this timestamp) –

    Info: Downloading EURUSD starting with
    WARNING: missing file http://www.dukascopy.com/datafeed/EURUSD/2008/00/01/00h_ti
    cks.bin (1199145600)

    • #1807 written by Tyler January 16, 2012 (2 years ago)

      Thanks for the confirmation, pipsaw. It’s been two days already…hopefully nothing permanent!

      • #1808 written by birt January 16, 2012 (2 years ago)

        Yeah, it’s been the weekend. They’re hopefully back to the office and already working on fixing the issue. As pipsaw mentioned, only data for the past 3 years is missing and the history downloader on the Dukascopy website is not working either, which probably means the whole thing is just a fluke, nothing permanent. Have some patience, I’m pretty sure it’ll come back up.

        • #1809 written by Tyler January 17, 2012 (2 years ago)

          Just to update everyone on the data available issue, while most pairs still seem to be down, I was able to get data for AUDNZD, EURAUD, EURUSD, GBPUSD, and GBPJPY last night. Hopefully the other pairs will be available again soon.

  • #1810 written by Mike January 16, 2012 (2 years ago)

    I just started the php process and after not too long I started getting a lot of missing file messages. I tried accessing the .bin file directory from a different network connection to see if maybe they blocked my IP or something along those lines, but I’m getting 404 errors from there too, even when I try accessing one of the files I was able to successfully download when I first started the process. It looks like they may have removed the datafeed. Can anyone confirm? Is this data not available in a zipped format online somewhere or would that be some sort of violation???

  • #1811 written by Mike January 16, 2012 (2 years ago)

    and of course I would read the comments above mine AFTER I posted it lol

  • #1812 written by Szilard January 17, 2012 (2 years ago)

    Hi there. Is that normal that if “use real spread (11)” and “use fix spread (20)” are appied to a M5 backtest after each other, then the results vary dramatically?
    I ran an EA on USDCAD 2011.01. fix 0.1 lots 1000EUR starting equity.
    The real spread version resulted +123 EUR in one month
    The 2 pip spread version resulted – 155 EUR in the very same month.
    Strange thing is that even trade openings differ in the two versions. The EA uses price levels as trade open condition, not indicators.

    Additionally strange thing is that in the real spread version no volumes indicator can be calculated. Volumes is 0 continously.
    The condition “if (iVolume(NULL,0,0) ==1 ) ” never gives true result.

    What could cause so big difference?

    • #1813 written by birt January 17, 2012 (2 years ago)

      If your EA uses the volume in any way, it can’t be tested with real spread because when real spread is enabled, the spread replaces the volume data.

      • #1814 written by Szilard January 17, 2012 (2 years ago)

        Wow, thanks, i did not know that. However this might not be the reason for the big difference between the two tests.

        • #1815 written by birt January 17, 2012 (2 years ago)

          Yeah, it’s probably what caused the difference. On the other hand, if your EA had better results with volume = 0, you might want to think about taking it out of the picture.

  • #1816 written by Meena January 18, 2012 (2 years ago)

    Have an account with Oanda so I have access to their tick data. Run Birt’s patch set properly. I have a file for EURUSD I have it named: eurusd 11-1-10 to 11-1-11. It is placed in Experts/file. I have followed the instructions but I get the message “cannot open input file”. mt4 build 409. Have tried both Oanda files… any idea what’s going on? the fxt file is in include folder..
    thanks…

    • #1817 written by birt January 18, 2012 (2 years ago)

      If it says “cannot open input file”, it means one of the following:

      • Your file was not named as the pair name (EURUSD.csv) and you did not type the file name in the input parameter.
      • You forgot to type the file extension (.csv, .txt or whatever it is) when typing the file name in the input parameter. Please note that Windows Explorer might be configured to hide extensions for known file types which might be confusing you into thinking it has no extension.
      • If the above do not apply, you did not copy the file to the correct location
  • #1818 written by Alex January 18, 2012 (2 years ago)

    Please, tell me how I can download only eur|usd data with “download_dukascopy_data.php” as I don’t need others like aud|jpy with which the downloading begins ?

    • #1819 written by Simon D January 18, 2012 (2 years ago)

      you need to comment out the other currencies at the beginning of the program

      • #1820 written by Alex January 18, 2012 (2 years ago)

        What program should I use to edit *.php file ?
        Should I use “;” in front of other currencies to comment a line ?

        • #1821 written by Simon D January 18, 2012 (2 years ago)

          use notepad or wordpad or any other text editor. You need to comment out like this:

          //”EURSEK” => 1175270400, // starting from 2007.03.30 16:00
          “EURUSD” => 1175270400, // starting from 2007.03.30 16:00
          //”GBPCHF” => 1175270400, // starting from 2007.03.30 16:00

          every currency apart from the one you want two slashes in front

  • #1822 written by Alex January 18, 2012 (2 years ago)

    Ok, thanks. I just deleted all other currencies, now eur|usd is downloading.

  • #1823 written by Meena January 18, 2012 (2 years ago)

    address: C:\Program Files (x86)\OANDA – MetaTrader\experts\files

    file name: eurusd 11-1-10 to 11-1-11.csv

    Window 7

    Am sure I have the files placed and named correctly.
    If it is explorer blocking csv how do I fix that?
    thanks…

    • #1824 written by birt January 18, 2012 (2 years ago)

      If you’re using Vista, search the comments for a solution.

      If not, I would recommend renaming the file to EURUSD.CSV and trying that way. You might be mistyping its name or something like that.

  • #1825 written by Meena January 18, 2012 (2 years ago)

    Renaming the file does not work.
    In the past I have run Dukascopy successfully but it was in 2010.
    I have an old setup for FXDD from 2010. I’m going to try to install it without updating it and see if that will work with Oanda data.
    Maybe it is mt4 build 409 that is the problem.
    Trying to run this on Windows 7.
    I am sure I am doing everything as instructed.
    I can also try to run this on Vista on another computer.
    Thank for your patience..

    • #1826 written by birt January 18, 2012 (2 years ago)

      Don’t bother trying with Vista.

      Anyway, the problem might be caused by the Windows UAC. I suggest making a copy of your MT4 folder somewhere else and trying from there – for instance you could put the whole folder on the desktop, there are no such issues there.

  • #1827 written by Meena January 18, 2012 (2 years ago)

    Yikes. Sure would like to know what the problem is.
    Folder is moved to desk top.
    csv file is named EURUSD.CSV and is in experts/file folder
    setup FXDD build 228.
    Ran Birt’s patch on EURUSD
    tried both Oanda scripts
    still have message cannot open file?
    windows 7.

  • #1830 written by Meena January 18, 2012 (2 years ago)

    yes it is in experts/file.
    not experts/files

    I am trying to Google for discussion about similar problems…
    If I try to open the CSV file with notebook it says stopped working, would that just be because is it a large file 471020KB and it will take some time for it to open in notebook? Is it too big for notebook to open?
    The file in question is Oanda separated by commas.

    • #1831 written by Harald January 18, 2012 (2 years ago)

      you need special editors to open such large files. but this is not your problem.
      you wrote:
      yes it is in experts/file.
      not experts/files

      That’s WRONG! It must be in expert/files

      • #1832 written by birt January 18, 2012 (2 years ago)

        Yes, Harald is 100% correct.

  • #1833 written by Meena January 18, 2012 (2 years ago)

    notebook gives message (not responding) not stopped working.

  • #1834 written by Harald January 18, 2012 (2 years ago)

    Hi birt,
    at the next opportunity, can you fix a small issue in the php scripts. Windows-php does not support several date/time modifiers, like %D and %T, giving a warning.
    > gmstrftime(“%D %T”,$firsttick)
    you can replace it by
    > gmstrftime(“%Y-%m-%d %H:%M:%S”,$firsttick)
    if you like.

  • #1835 written by Meena January 18, 2012 (2 years ago)

    The csv file is in the proper folder. My message on the thread has a wpelling errors. sorry about that But the csv file is in the proper folder. That is not the porblem.
    I have doenloaded the Process Manager. And filtered for terminal.exe.
    How to find the log?

  • #1836 written by Jason January 18, 2012 (2 years ago)

    Birt

    I seem to be having problems when running optimisation after running the Brits Patch. I just wanted to check the following in the Experts journal is correct:

    2012.01.18 22:32:18 birt’s patch EURUSD,M15: removed
    2012.01.18 22:32:18 birt’s patch EURUSD,M15: uninit reason 0
    2012.01.18 22:32:18 birt’s patch EURUSD,M15: Process patched for variable spread at 0×557016. Volume check removed at 0x55734E.
    2012.01.18 22:32:17 birt’s patch EURUSD,M15: Patched: fseek 0×557244, fopen 0×557154, fclose 0x55710E, fclose 0x555BD5, fclose 0x5563EE, fread 0x55729A, filelength 0x55718D. 2gb limit removal patch was installed at 0×4270000.
    2012.01.18 22:32:17 birt’s patch EURUSD,M15: FXT overwriting disabled. Addresses patched: 0×556122, 0x55623B, 0×556343.
    2012.01.18 22:32:16 birt’s patch EURUSD,M15: MT4 build 409 detected.
    2012.01.18 22:32:16 birt’s patch EURUSD,M15 inputs: NoFXTOverwrite=true; Remove2GBLimit=true; WARNING2=”Using variable spread with a non-variable spread FXT does NOT work.”; RealSpread=true;
    2012.01.18 22:32:06 birt’s patch EURUSD,M15: loaded successfully..

    Jason

    • #1837 written by birt January 19, 2012 (2 years ago)

      Optimization does not work with that script with build 405+ and it has problems even with previous builds.

      I am working on a loader that also supports optimization, but it’s going to be launched as a commercial product. It’s already been in beta testing for a while. Keep a close eye on the site during the following couple of weeks if you are interested.

  • #1838 written by Meena January 19, 2012 (2 years ago)

    I tried a couple of things which seem to have fixed the problem of “cannot open the file” in windows 7 for Oanda build 409. In permission I disabled “WMPNetworkSvc”. Named the file simply EURUSD, although I had tried to run the script before with that name with no success. Anyway the file is converting now. Thank you…

  • #1839 written by Meena January 19, 2012 (2 years ago)

    I have the file converted to hst and they in the proper folder The fxt is in its proper file. I opened the terminal ran Birt’s patch, then built on the EURUSD chart. As soon as tried to run the tester the platform closed.

    • #1840 written by birt January 19, 2012 (2 years ago)

      That can be caused by a number of things:

      • You’re running a recent EA on builds 402 or earlier
      • You have run another backtest prior to running the Birt’s patch script
      • You’re trying to run an optimization.

      Judging by what you wrote above, you are running version 228 which will crash with pretty much every other EA available nowadays. I suggest trying to attach the EA to a chart before trying to backtest it. If your terminal build is 402 or below, consider trying with build 409.

      Other options to try: restart the terminal, run the birt’s patch script as soon as it starts, attach the MACD EA to a chart and if that works, backtest it.

      As for your previous problem with the CSV file, it was most likely caused by the fact that you named it EURUSD.CSV.CSV but windows explorer was hiding the file extension. Google “hide extension for known file types” if you wish to disable that.

  • #1841 written by Alex January 19, 2012 (2 years ago)

    I reloaded tick data with php, again followed all the instructions so everything is in its place, run the patch script; but when I try MACD_Sample or any other (it skips the “Collecting M??” and “Using M??” steps) I get 25% modeling quality. I decided to check the data with the “MT4i Data Integriry Checker”, it says that data is OK. How can I finally get 99% ? Help.

    • #1842 written by birt January 19, 2012 (2 years ago)

      I would venture a guess that you are using the “control points model” instead of “every tick”.

      If that’s not it, check that you have DLL calls enabled (the FXT should be impossible to overwrite because it should be set as read-only if you had that enabled when creating it) and if you do, please post the lines related to the birt’s patch script that appear in your experts log after running it.

      • #1843 written by Alex January 21, 2012 (2 years ago)

        I use “every tick”.
        Here is the log:
        “”
        13:16:33 birt’s patch GBPUSD,H1: loaded successfully
        13:16:43 birt’s patch GBPUSD,H1 inputs: NoFXTOverwrite=true; Remove2GBLimit=true; WARNING2=”Using variable spread with a non-variable spread FXT does NOT work.”; RealSpread=false;
        13:16:43 birt’s patch GBPUSD,H1: MT4 build 409 detected.
        13:16:43 birt’s patch GBPUSD,H1: FXT overwriting disabled. Addresses patched: 0x5560E2, 0x5561FB, 0×556303.
        13:16:45 birt’s patch GBPUSD,H1: Patched: fseek 0×557204, fopen 0×557114, fclose 0x5570CE, fclose 0x555B95, fclose 0x5563AE, fread 0x55725A, filelength 0x55714D. 2gb limit removal patch was installed at 0×3840000.
        13:16:45 birt’s patch GBPUSD,H1: uninit reason 0
        13:16:45 birt’s patch GBPUSD,H1: removed
        “”

        • #1844 written by birt January 22, 2012 (2 years ago)

          If you still get 25% check that you actually generated the FXT for the timeframe you are testing. Could be that you’re backtesting using M1 and the FXT was generated for another timeframe.

          • #1845 written by Alex January 23, 2012 (2 years ago)

            Hi, Birt.
            I am actually using m1 and the FXT file is generated for m1 also. I am confused, don’t know what to try now.

          • #1846 written by birt January 23, 2012 (2 years ago)

            I’m not sure what could be wrong but getting a 25% modeling quality on M1 means one of the following:
            - you are using control points
            - you are using regular backtesting (interpolated ticks)

            This means the FXT you generated was not used, either because it wasn’t copied to the correct location or because it was overwritten when you started the backtest. Judging by the log you posted you seem to have correctly run the script (I’m assuming prior to the backtest), so I would check that the tick data FXT is in tester/history. Copy it once more if you have to because if you got 25% once, you must have a 25% percent FXT in there. Another thing to make sure is that you’re copying it to the correct MT4 folder if you have multiple installations on your computer.

            • #1847 written by Alex January 24, 2012 (2 years ago)

              I don’t know how but now it works and shows 99%. I have done all the same.
              Thanks for your help.
              When I test for m1 the horisontal bar under the modeling quality is red, is it ok ?

              • #1848 written by birt January 24, 2012 (2 years ago)

                Yes, that’s normal with tick data backtests.

  • #1849 written by Jason January 19, 2012 (2 years ago)

    Birt

    Is there a Script that works with Build 409 that I can download?

    I’m also up for any beta testing ;-)

    Jason

    • #1850 written by birt January 19, 2012 (2 years ago)

      The script that is available for download for about 2 months now works with build 409 with a few limitations (it must be started prior to any other backtest and optimization does not work).

      As for beta testing, one of the requirements is prior experience and a certain degree of familiarity with tick data backtesting. If you feel you meet this criteria, please get in touch with me using the contact form.

  • #1851 written by John Bennett January 20, 2012 (2 years ago)

    Bert – Awesome blog! The Dukascopy loader is better than a twelve pack, and maybe NFL football (go Giants!). Thanks, and I will check in often.

    John, Currency and equity trader

  • #1853 written by Tyler January 21, 2012 (2 years ago)

    I’m still getting 404 errors on most pairs for recent tick data. It’s been over one week now. Can anyone confirm they are seeing the same thing? If anyone is a Dukascopy client, do you know if Dukascopy has said anything about this? I appreciate any feedback anyone may have. Thanks.

    • #1854 written by birt January 21, 2012 (2 years ago)

      I have the same issue. I heard no news from Dukascopy but I didn’t even try to keep an eye on their channels. I suppose it’s going to be restored at some point, they may have had some problems.

    • #1855 written by Alex January 22, 2012 (2 years ago)

      Just to contribute something as well (even if little): I wrote an email to Dukascopy and they confirmed that the download issue is just a temporary thing and they are working on getting it fixed.. so it should be up again soon.

      • #1856 written by birt January 22, 2012 (2 years ago)

        Good to know, thanks! :)

      • #1857 written by Tyler January 22, 2012 (2 years ago)

        Yes, thanks for confirming, Alex. I appreciate it.

  • #1858 written by Jak January 21, 2012 (2 years ago)

    what do i type.. jsut to download EURUSD… with tht php method

    php download_dukascopy_data.php

    or lets say only EURUSD and USDJPY

    Thanks a lot

    Jak

    • #1859 written by birt January 21, 2012 (2 years ago)

      Please scroll up to comment #1815 just above and read that thread.

  • #1860 written by Jason January 22, 2012 (2 years ago)

    Does anyone have a build of MT4 that works with the Optimisation Script?

    I can only find Build 409…I have build but I’m not sure if it needs to connect to the broker or can you use it with the converted tick data without being connected.

    • #1861 written by Alex January 22, 2012 (2 years ago)

      209 works fine. I just downloaded it again, google will find it in some forum post..

  • #1862 written by Mike C. January 23, 2012 (2 years ago)

    Hey Birt, what hex editor did you use to edit the .fxt file? Is there an easy way to modify the leverage like there is the spread? Thanks,

    Mike

    • #1863 written by birt January 23, 2012 (2 years ago)

      Personally, I use hiew because it can also disassemble stuff but I wouldn’t recommend that one to you because it’d feel clumsy. Just get any windows hex editor, e.g. winhex.

      You can edit the leverage at offset 0×164 I think (or somewhere close to that, it should be 0×64 if you set it to 1:100, just look for that and change it).

      • #1864 written by Mike January 26, 2012 (2 years ago)

        Awesome, I found the field and the corresponding value. I had actually set mine to 200, so it was C8. I’d like to change it to 300, but 300 in hex uses three digits…should it overlap into the field on the left or the right of C8? If you’re not sure, no big deal, just trying to see if I can save time from generating a new FXT file with different leverage if I don’t have to. Thank you!
        Oh by the way, found an amazing hex editor called Neo Hex Editor, looks up the whole 3gb file in an instant and is very browsing friendly

        • #1865 written by birt January 27, 2012 (2 years ago)

          Into the field on the right. It should look like 2C 01.

  • #1866 written by Jason January 23, 2012 (2 years ago)

    Alex

    I’m assuming that if your using Tick Data there is no need to be connected to a broker first before running any optimisation?

    Jason

    • #1867 written by Alex January 23, 2012 (2 years ago)

      Well to convert the data via the JForex2FXT script you need to be connected. But I guess a demo account should be sufficient. The 209 build I found was customized for Sunbird, but I just copied my Alpari setting into it.. Later on you don’t need to be connected anymore. Do you know any security issues with this build?

  • #1868 written by SvenGali January 23, 2012 (2 years ago)

    Hi Birt,

    getting data from Dukascopy works really fine – great blog / great work – thx! I have an issue with JForex2FXT (FXPrimus Metatrader Build 409). The EUR.USD csv-file looks good (complete – checked with ultraedit), but when I apply the script to the 1M-chart, it only reads data until mid of 2010 (exactly 2010.06.09 19:36), then starts shortly with 2012 (but only weird data, i.e. 2012.12.31…) and then stops. If I load the according hst-files, I can see that most of 2010 and 2011 is missing and I think it’ll be the same with the fxt-file. I already tried to use a smaller time-frame but that doesn’t solve the problem. The file-size of the csv-file is 5.8 GB. Could you tell me more ’bout the possible cause? Thx in advance. SvenGali

    • #1869 written by birt January 23, 2012 (2 years ago)

      There’s an error in your CSV file. I’ve already written a script that is able to handle such cases over a week ago, but it changes things dramatically and I have to rewrite that part of the guide, plus the downloading part which I’ve already rewritten but not published yet. In other words, the script is ready but it is waiting for me to have the time to rewrite the guide, which might happen this week or perhaps the next.

      In the meantime, I suggest reading the last couple of comment pages for solutions as you’re not the only one that ran into this problem.

  • #1870 written by Jason January 24, 2012 (2 years ago)

    Alex / Birt

    I’ve managed to find a Sunbird build 229 and got it to connect to the broker using the login from a version 409. Converted the CSV and transfered the files but still no luck getting the MT4 to run a Optimisation.

    I’m converting the FXT using a 2.0 pip spread.

    Any suggestions?

    I’m running the MT4 on Windows 7

    Jason

    • #1871 written by birt January 24, 2012 (2 years ago)

      no luck getting the MT4 to run a Optimisation.

      That’s not very descriptive. How does it behave? Does it crash? If so, did you try your EA on a chart in build 229 prior to trying it in an optimization?

    • #1872 written by Alex January 24, 2012 (2 years ago)

      I use XP and it all works fine.

  • #1873 written by Jason January 24, 2012 (2 years ago)

    Birt

    I think my issue is with the tick data once converted…something is going wrong. I’ve tried both JForex & Dukas Copier CSV files and converted with the corresponding scripts to product both FXT & HST files then copied to the directories. I then run the Birts script and start the Optimisation, but I’m not getting any Opimisation results – i’ve tried several EA’s including the sample MACD sample. When i convert the CSV files shoulds i input both the Spread which I’ve been setting to 2.0 and also the Real Spread? or just select one or the other? I also found the that the newer MQL script makes the 229 build crash, so i’ve used an older Script I found in a post from last year…

    I’m just converting 1 year of tick data at a time, and tried leaving the Use date box ticked and unticked..

    Jason

  • #1874 written by Jason January 25, 2012 (2 years ago)

    Birt

    I forgot to add that i’m converting 1 year of data and I can get the MT4 to run only on 1 month of data, and it goves me optimisation results, any longer and it the Journal is saying : 2012.01.25 01:03:25 There were 4 passes done during optimization, 4 results have been discarded as insignificant…..

    Jason

    • #1875 written by birt January 25, 2012 (2 years ago)

      Discarded results means the runs were not profitable. You can disable discarding results by right clicking the optimization results window and enabling an option in that menu.

  • #1876 written by Jason January 25, 2012 (2 years ago)

    Birt

    My back tests run with 99% modeling quality and use the full years data, but only 1 month when optimising

    Jason

    • #1877 written by birt January 25, 2012 (2 years ago)

      Optimization is not supported with the Birt’s patch script. If you can get it to run, good for you; otherwise, if you need to run optimizations you can just wait for the Tick Data Suite which is now in beta testing and which should be released within the following couple of weeks.

      Basically, the script was never supposed to work with optimization although it seems to occasionally do so with builds older than 402 but with severe limitations (such as only selected periods). Please read the previous comment pages if you need any more information.

  • #1878 written by jore January 25, 2012 (2 years ago)

    Hi birt,

    I got the following messages when create fxt files with Dukascopy2Fxt.
    I create csv-files with dukasCopier.

    2012.01.25 10:01:25 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2029.
    2012.01.25 10:01:22 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2027.
    2012.01.25 10:01:21 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2012.
    2012.01.25 09:56:48 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2009.
    2012.01.25 09:34:15 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2008.

    But no such years are in the csv-file, I checked it with uedit.
    The fxt file is relativly small (1,454,540kB), when I try it, MT4 crashes.

    I use MT4.209 Dukascopy2fxt from 11.03.2011.

    In the log of DukasCopier there are some “Zero length file skipped.” messages, but I think that is OK.
    And it finished with:
    EURUSD\20120\23\23h_ticks.bin – Der Remoteserver hat einen Fehler zurückgegeben: (404) Nicht gefunden.

    Weeks ago all things work well.
    Any ideas?

    Thanks

    • #1879 written by birt January 25, 2012 (2 years ago)

      It’s an issue with the Dukascopier. Please read the last couple of comments pages for more info.

    • #1880 written by Xman January 27, 2012 (2 years ago)

      I had the same problem with incorrect years like 2029, 2031 showing up in the FXT only.

      Here’s a workaround :
      In my case, the issue seems to happen only when my csv files were over 4gb. So I used the “gsplit” freeware tool ( http://www.gdgsoft.com/gsplit/ ) to split my CSV into 2 files.

      My EURUSD 2007-2011 CSV filesize was 6.8gb and with gsplit configured to split the csv file after 67million linefeeds characters, it did the job nicely. It resulted in 2 files, first file had data from 2007/04 to 2008/11, second file had 2008/12 – 20011/06. I took a peak at the first few lines of the csv by using BeyondCompare and clicking stop before it loads the entire file and max out your ram. If there is a better text editor to stream view these massive files, I would love to know.

      Renaming the second file to EURUSD.CSV and re-running Dukascopy2fxt did the trick. No more invalid 2029,2031 years in the log. The caveat is you can only load one of the csv files into the FXT, I loaded the second file (2008/11 to 2011/06), the first file can be loaded separately (replacing current FXT) if needed later.

      Similar process worked on a Jforex file 2007-2012, split it so last file is under 4gb and run the converter. Its a workaround until MT4 or coverter can support over 4gb csv files.

  • #1881 written by Jason January 25, 2012 (2 years ago)

    Birt

    Looking forward to the Tick Data Suite…

    Keep up the great work..

    Much appreciated

    Jason

    • #1882 written by birt January 25, 2012 (2 years ago)

      If you’re the Jason P. who contacted me on the 21st, please contact me from a different email address as I haven’t been able to reply to the one you provided.

  • #1883 written by nicco January 25, 2012 (2 years ago)

    I can confirm Ticks have stopped download on JAN 23 @ 12h

    Missing Files from then on…

    Birt and ideas/New from Dukas?

    • #1884 written by Harald January 25, 2012 (2 years ago)

      it is Jan 23 2012, right?
      this is unrelated to the current problem that long periods of some currencies are missing.
      In Dukas data quite often the last 2-3 days are missing, but will be filled it a couple of days later.

      • #1885 written by Harald January 29, 2012 (2 years ago)

        just checked: Now all data is there until Jan 27, 8 a.m. Rest of 27 is missing.

  • #1886 written by Delbru January 25, 2012 (2 years ago)

    Hi Birt,

    For you info I want to back test my EA on tick data from dukascopy using the ‘JForex2FXT’. Data range is 09/01/2011 to 12/30/2011. I followed all the steps as you mentioned.
    When I ran the ‘birt’s patch’ from scripts I got the following error.

    2GB limit removal not activated.

    If you’re using Windows XP, consider getting a copy of the Visual C 2010 runtime, available at http://www.microsoft.com/download/en/details.aspx?id=5555 (x86) and http://www.microsoft.com/download/en/details.aspx?id=14632 (x64).

    Could not find the _fseeki64() function in your msvcrt.dll or msvcr100.dll!

    Also in the Forex.com Metatrader strategy tester I get the error for 5m and any other time frame.
    TestGenerator: no history data ‘XAUUSDFXF5′

    • #1887 written by birt January 25, 2012 (2 years ago)

      If you only wish to backtest such a short time period, you don’t even need to care about the 2GB limitation. The FXT files will be larger than 2GB only if they span several years.

      As for fixing the 2GB problem, if you’re using MT4 build 405 or higher, all you have to do is what it tells you: download the MSVC10 runtime.

      Finally, regarding the “no history data” message, that means that your history center does not have history for the XAUUSDFXF5 pair for the timeframe you selected. Try opening a chart with that pair and timeframe and keep pageup pressed for 30-60 seconds.

      • #1888 written by Delbru January 27, 2012 (2 years ago)

        Hi Birt,

        I Downloaded 3 months worth of XAUUSD Tick data from 09/01/2011 to 12/30/2011.
        I placed the XAUUSDFXF in Forex.com MT4 ‘experts/files’ folder. I am not sure how to view chart of this particular dataset. Whenever I drag and drop the XAUUSDFXF from ‘navigator’ I only see the most current charts. My EA is sort of medium frequency algo which reads incoming tick flows and tries to predict price for a future horizons. The candles 5m, 15m, are only used for the EA to be able to restart itself everytime…. and also also for some post trade order management. My ea doesn’t use O,H,LC values of any candle either. My objective is to carry out an in-sample test my EA on this 3 months tick data. How do I do that?

        • #1889 written by birt January 27, 2012 (2 years ago)

          If you wish to backtest it, all you have to do is follow the guide above. Basically, you have to convert your XAUUSD CSV data to an FXT + HST files, copy them to the correct locations and proceed with the backtest. If you want to view the resulting chart, just open a chart after you copied the HST files.

          • #1890 written by Delbru January 28, 2012 (2 years ago)

            Hi Birt,

            I was finally able to back test some of my signals, but, am still not certain about the quality of my test. For example within my EA is a data recorder which logs data at entry and exit of trades things like prevailing (bid/ask), past few quotes (bid/ask) from the time of the trade, Volume[0] and time of trade ‘Current_Time’ in millisec etc.. into a csv file named after that day. Now, when I ran the back-tests for 5m and 15m after creating .fxt and .hst files (and placing them in the right folders) for 5 days i.e. from 09/02/2011 to 09/07/2011 no such csv file was created. I also felt that the trade data was not very granular i.e. for example if at time 00:00 there is a exit signal for a long or short position the actual exit happened 5 to 7 minutes later. where as when I run the EA live the exit happens almost instantly. Are there ways to make this back test more accurate and that emulates the real-time run? Thanks!

            • #1891 written by Harald January 29, 2012 (2 years ago)

              Hi Delbru,
              in my experience, it is very accurate in simulating the realtime.
              If you observe strange behaviour, why don’t you use the visual mode? You can pause and run it in slomo, a tick per second if you want. put some more debug messages in your EA and you should easily find out the problem

  • #1892 written by Neo January 26, 2012 (2 years ago)

    Birt, will your new Tick Data Suite allow for “Open prices only” Model back testing? It appears that currently the “Open prices only” model crashes MT4 whether running a single scan, or trying with Optimization.

    Thanks,
    Neo

    • #1893 written by Harald January 26, 2012 (2 years ago)

      Neo,
      just for curiosity: MT4 supports open-price backtesting, and with this method I feel “real” tick data is pretty useless. So what do you miss from what the MT4 platform is offering by default?

      • #1894 written by Neo January 27, 2012 (2 years ago)

        The only reason I ask is because the WFA has to do a complete Optimization back test before doing its walk forward tests…so they suggest (to speed up the first optimization back test), that you use the Open Prices only for that leg of the journey.

        Neo

        • #1895 written by Harald January 29, 2012 (2 years ago)

          Neo,
          this could lead to less good results. Either your EA needs tick data for realistic results or not. You should run both legs with the same method.

          • #1896 written by Neo January 30, 2012 (2 years ago)

            I know what you mean and you’re totally right…but the WFA people recommend using the Open Prices Only model for the optimization scan that is done before the application then runs the scan again doing walk forward analysis. The only info they need from the initial Optimization test are the parameters used in the EA…accuracy is not important. Then the WFA app takes that info and feeds it into its application that then does the optimization scan over again. Using the Open Prices only for the first step saves time.

            Kind Regards,
            Neo

    • #1897 written by birt January 26, 2012 (2 years ago)

      Harald is correct, you don’t really need the patch or tick data if you want to use control points or open prices only. The CSV parsing script does not produce an FXT that can be used with these models.

      However, using the Tick Data Suite will no longer crash so in that respect I guess you could say it works.

  • #1898 written by Pawel January 26, 2012 (2 years ago)

    GMT offset for tick data from dukascopy is 0 , what is UTC offset for this data?

    • #1899 written by birt January 27, 2012 (2 years ago)

      GMT is pretty much the same as UTC, so the UTC offset is 0. Google GMT vs UTC if you need more info on the topic.

      If you are asking if the data is shifted by 1 hour during the summer, it isn’t.

  • #1900 written by www.fxfir.com January 27, 2012 (2 years ago)

    hello
    do you have direct download mt4?
    very thank you

  • #1902 written by Neo January 27, 2012 (2 years ago)

    I don’t do Open Price only back tests, but the WFA needs to do a complete Optimization back test first, before doing its analysis…so in order to speed up the first scan, they suggest doing Open Prices only.

    Neo

    • #1903 written by birt January 27, 2012 (2 years ago)

      The Tick Data Suite will work fine with all backtesting models, although FXT overwriting is only disabled for every tick backtests.

  • #1904 written by Jane January 27, 2012 (2 years ago)

    Hi Birt,
    I can backtest my EA with Dukascopy Tick Data now, millions thanks !!!
    Another question please, you mentioned user can run MT4 EA on platform of JForex with a very tricky way, can I know what’s that? Because I really want to find a stable ECN like Dukascopy, thank you!!!

  • #1906 written by KlamseN January 27, 2012 (2 years ago)

    Hi birt, any idea why the 2gb removal doesn’t work for my new laptop?
    It’s a Win7 64bit maschine with 8gb ram.

    2012.01.27 19:39:33 birt’s patch EURUSD,H1: uninit reason 0
    2012.01.27 19:39:33 birt’s patch EURUSD,H1: Process already patched for the 2gb limit removal or we just can’t find the area to patch.
    2012.01.27 19:39:33 birt’s patch EURUSD,H1: You should check for a new patch at http://eareview.net/tick-data
    2012.01.27 19:39:33 birt’s patch EURUSD,H1: The patch you are running was not tested with this build so it may or may not work.
    2012.01.27 19:39:33 birt’s patch EURUSD,H1: MT4 build 409 detected.
    2012.01.27 19:39:33 birt’s patch EURUSD,H1 inputs: NoFXTOverwrite=false; WARNING1=”The 2GB limit removal only works in Windows 7 and Vista.”; Remove2GBLimit=true; WARNING2=”Using variable spread with a non-variable spread FXT does NOT work.”; RealSpread=false;
    2012.01.27 19:39:18 birt’s patch EURUSD,H1: loaded successfully

    I already tried to load the patch prior to everything else, but still the same problem. And I still have the limitation of 2gb :-/

    Thanks for helping and best regards

    • #1907 written by birt February 1, 2012 (2 years ago)

      You are using an old version of the script. Please download the latest.

  • #1908 written by Marcel January 29, 2012 (2 years ago)

    Hi Birt, since the upgrade to build 409 of Metatrader I am not able anymore to use the downloaded tickdata. I think it’s the same kind of issue Meena is experiencing at the moment but I am pretty sure the suggested mistakes are not the reason for this error. It has something to do with build 409 but I don’t know what, I hope you can help me.

    I have two different version of Metatrader installed, build 402 and build 409. I have downloaded the tickdata, installed them in exactly the same directories and then ran Dukascopy2FXT and that’s where it goes wrong because when I look at the expert messages of build 409 I see that it’s starting to process i.e EURUSD 2007, 2008 and 2009, but then it jumps to 2012, 2027, 2028, 2029,2030 and 2031. Build 402 is processing the same csv-file correctly without any errors.

    After execution of the Dukascopy2FXT-script I copy the fxt- and hst-files to the correct subdirectories (the size of the build 409-files are appr. half of the size of the 402-files) and start a back test in both builds. Build 402 is working fine and build 409 is giving the error: TestGenerator: internal error because the file could not opened.

    I am pretty sure it has something to do with the new build. Do you have any idea what it is and if there will be a solution for this problem soon?
    Thanks by the way for all the work you have done so far, much appreciated!

    • #1909 written by birt February 1, 2012 (2 years ago)

      “TestGenerator: internal error because the file could not opened.” means that you forgot to apply the Birt’s patch script.

      I will look into generating the FXT with build 402 and 409 sometime this week and check if that’s what’s causing the errors.

      • #1910 written by Marcel February 1, 2012 (2 years ago)

        Hi Birt, thanks for your reply. I have definitely applied the Birt’s patch script. I have downloaded the latest version of the scripts, but I still get the same message. I hope you can solve this issue.

        • #1911 written by birt February 2, 2012 (2 years ago)

          When you apply the patch some lines are printed in the experts log. Please paste them here.

  • #1912 written by KlamseN January 31, 2012 (2 years ago)

    Hey birt,

    is that a known problem? I would be very happy if you could help me on that…
    Thanks and regards

  • #1913 written by Steve February 2, 2012 (2 years ago)

    I already have history files in my “history\your_server_name” folder. Do you really mean put them in “tester\history” folder? If I overwrite the history files in “history\your_server_name” I get “OldTick” errors and a flat and/or spikes in the chart when I restart MT4.
    Thanks

    • #1914 written by birt February 4, 2012 (2 years ago)

      If you get that problem, you likely had the issue with skipped years when generating your files.

  • #1915 written by kain February 2, 2012 (2 years ago)

    Hi Birt,

    Sorry if this sounds stupid, I used your 1min candle script to download 1 min data, and I tried to follow the steps for converting .csv to .hst and .fxt files for mt4. Even
    though I got them converted for 1 min and 1 hour, when I looked at the charts in offline mode, the whole screen appears greeen with bars. This is for AUSUSD. Does your script only work from tick data. If so, do I need to modify the script to accept 1 min data as the minimum for higher conversion.

    I only wish to back test on opening and closing of 1 min data and higher and do not require tick level precision at this moment. I wish to manually test the program as such I need to be able to see data as far back as possible.

    I would be happy with any help on this…
    Thanks and regards

    • #1916 written by birt February 4, 2012 (2 years ago)

      The processing script only works with tick data. For candle data, you need to make the import process work on your own (aka write some code for it). I have a tentative script that does that but I gave up quickly on it due to the volume issues. I might decide to publish it as well at a later time but don’t count on that happening too soon because I am extremely busy these days.

      • #1917 written by Kain February 5, 2012 (2 years ago)

        Thanks for the heads up Birt, I think I have to read your codes throughly and modify them.

        Will inform you how things are going once I have tested them.

        Thanks and regards.

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top