How to backtest using tick data with Metatrader 4

If you’re trying to follow this guide, you must have gone through acquiring tick data and converting it for Metatrader 4 usage and as a result you should have the HST files in your history\[your broker's name] folder and the FXT in your tester\history folder. If you’re not at this stage, please revise the guides linked above.

This guide is split in several sections:

  1. Enabling tick data backtesting
  2. How to verify it worked
  3. Running multiple Metatrader 4 instances from the same folder

Enabling tick data backtesting

Since the recent Metatrader 4 client builds no longer offer the ability to use custom FXT files, you need to run a special software that enables it to do that.

At this point, you have two options:

  • You can use the Tick Data Suite, which is a commercial (aka paid) software that has more features than the free script and is fully supported. A trial version is available.
  • Alternatively, you can use the completely free Birt’s patch script which has some limitations*. No support is available.

*The limitations that apply to the free script include but are not limited to:

  • You cannot run optimizations (trying to do so results in a crash).
  • It does not work with third party tools such as the Walk Forward Analyzer.
  • You must run the script manually every time you restart Metatrader 4.
  • The 2GB limitation removal is only available for builds 405+ and for select operating systems for previous builds.
  • If you run the script after performing a backtest without it, a subsequent backtest will result in a crash.
  • You have to manually enable real spread and you have to restart the terminal if you no longer want to use real spread.
  • There is no support. You have to make do with the older comments on the tick data page and with the troubleshooting section.

For a more in-depth comparison table, please head to the Feature matrix page.

Depending on your choice above, please follow the relevant guide:

How to verify it worked

If your backtest is running and the file in tester\history was not recreated, it usually means everything is ok. Naturally, it’ll skip the “Collecting M??” and “Using M??” steps. At the end of the backtest, you should see a red bar (it’s normal) in the results tab and modeling quality 99% in both the results tab as well as on the balance chart.

If you run into any problems, please head to the FAQ & Troubleshooting page.

Running multiple Metatrader 4 instances from the same folder

If you’re using the Tick Data Suite, this is done automatically and you can start as many MT4 terminal instances you wish from the same folder.

Otherwise, if you are using the Birt’s patch script and you need to run multiple instances from the same folder you can use one of the loaders available in the loader archive on the tick data downloads in the Miscellaneous section. These loaders are working with builds 225-409; the generic loader in the main folder works with builds 405-409 while the generic loader inside the old folder will work with older versions. There are also individual loaders for several older builds but the generic loaders should work for pretty much everyone as long as the build used is not above 409. Please note that no support is available for these loaders.

Keep in mind that the generic loaders will need to configure themselves for your particular build so they will display a series of messages and close your terminal once the relevant offsets are found. This only happens the first time you run them and it is perfectly normal.

  • #1 written by nicco February 9, 2012 (3 months ago)

    Hi Birt,

    with your new fxt scripts – for the first time I am getting a GREEN bar on the backtesting modeling quality reports page!

    makes me worry about 2011 – Jan 2012. I only ever had a red bar on the modeling quality.
    Whats the difference/improvement here?

  • #2 written by nicco February 9, 2012 (3 months ago)

    sorry – or course reading helps #6 above.
    but still the question: whats the difference?

    • #3 written by birt February 9, 2012 (3 months ago)

      Just 1 byte in the FXT. There’s no difference at all other than the fact that it’s green instead of red. There were too many people asking about it so I decided to fix it even though it was merely a cosmetic issue.

  • #4 written by Keith February 9, 2012 (3 months ago)

    Birt,

    I have an issue using pepperstone mt4 and dukascopy data. downloaded all the data using dukascopier, created the hst and fxt from the csv generated using real spread. ran the new patch with real spread enabled. when i ran my EAs it keeps telling me that the lot size is smaller than broker min lot size. but i know that pepperstone can be backtested using 0.01 lots on normal metaquotes data. using 0.1 lots solved the problem of not trading but the lot size is totally out.

    Can you check whether it’s a broker problem or data problem?

    • #5 written by birt February 10, 2012 (3 months ago)

      It’s completely unrelated to the tick data. The minimum lot size is set to the minimum lot size of the broker that you are connected to when generating the FXT.

      Check it out by creating a script that contains the following in its start function:
      Comment("MODE_MINLOT: " + DoubleToStr(MarketInfo(Symbol(), MODE_MINLOT), 2));
      Run the above script on the same chart you ran the conversion script (note that the min lot might be different from one pair to another).

      • #6 written by muddog February 10, 2012 (3 months ago)

        I believe I see where the problem is. the commission account i logged in seems to be 0.1 and the actual trading account is 0.01. it’s a pepperstone problem

      • #7 written by muddog February 10, 2012 (3 months ago)

        birt if recreating the fxt files on a different login but same server, do i have to recreate the hst files? or are the hst creation independant of the fxt creation in this sense?

        • #8 written by birt February 10, 2012 (3 months ago)

          There’s no need to recreate even if you switch servers. You just have to copy the HST files in the server’s history folder.

          You only have to recreate the HST files if you change the start/end time, the GMT offset or the DST.

  • #9 written by Dan February 17, 2012 (2 months ago)

    Hi Birt, my MT4 crashes everytime I tried to run optimization using tick data. Have you experienced this problem before?

    Here’s the crash log:

    Filename : mt4clw.B9171DE9253A2D8A37410015A346A6B7
    Time : 2012.02.18 00:30
    Program : MetaTrader 4 Terminal
    Version : 400.409 (25 Oct 2011)
    Revision : 2771
    OS : Windows 7 Professional 6.1 Service Pack 1 (Build 7601)
    Processors : 4 x Intel Core i3 M 350 @ 2.27GHz
    Memory : 2242 free of 3950 Mb
    Virtual : 1856 free of 2047 Mb
    Exception : C0000005 at 77EA8DC9 write to 00000014

    Modules : 00400000 00785000 terminal.exe
    : 71AE0000 00027000 wlidnsp.dll
    : 71A50000 00021000 mdnsnsp.dll

    77EA8ACD:0002FC [77EA8DC9] RtlIpv4AddressToStringExW (ntdll.dll)
    77EA8ACD:00020B [77EA8CD8] RtlIpv4AddressToStringExW (ntdll.dll)
    0058AA99:00002E [0058AAC7] #6694 (terminal.exe)
    00556C80:000079 [00556CF9] #5638 (terminal.exe)
    0053ACC0:000A48 [0053B708] #5342 (terminal.exe)

    Registers : EAX=00000000 CS=0023 EIP=77EA8DC9 EFLGS=00010213
    : EBX=FFFFFFFC SS=002b ESP=04A4BCC0 EBP=04A4BD10
    : ECX=00000000 DS=002b ESI=76372980 FS=0053
    : EDX=00000004 ES=002b EDI=76372984 GS=002b

    • #10 written by birt February 17, 2012 (2 months ago)

      Please read section 2, question 5 in the FAQ. Short answer: you can’t run optimizations with the Birt’s patch script.

  • #11 written by Dan February 17, 2012 (2 months ago)

    Thanks Birt. Does your tick data suite support optimization?

  • #13 written by Dan February 17, 2012 (2 months ago)

    Sorry, pls ignore my question above.

  • #14 written by Ed February 24, 2012 (2 months ago)

    I am trying out the free trial.

    I want to collect Dukascopy tick data for multiple currency pairs from January 2011 – Feb 2012.

    Right now I am downloading tick data for EURUSD – but by the looks of it this alone could take from 10 – 20 hours just to download 1 years worth of tick data at Dukascopy for EURUSD.

    I have dsl wireless.

    Is it common for the downloading of tick data to be this slow?

    What happens if someone wants to download 4 years worth of tick data from multiple currency pars. It could take several months to download?

    Is there a way around this slowness?

    • #15 written by birt February 24, 2012 (2 months ago)

      It takes some 2-4 days for me to get all the data for all pairs with a good connection. I assume it’d take longer with a DSL and even longer if you’re using wireless. There’s not much you can do to work around it, I’m afraid. If you’re using a laptop, I suggest leaving it closer to the wireless router or using a wired connection while you download.

  • #16 written by Aletheia February 25, 2012 (2 months ago)

    Hi Birt

    I have reinstalled everything and now get 99% data reported.

    Unfortunately, there are no results! I look in the log and it says:

    2012.02.25 18:32:27 2011.06.01 01:00 xEAx EURUSD,M15: EA does not work!
    2012.02.25 18:32:27 2011.06.01 01:00 xEAx EURUSD,M15: The number of bars is not enough
    2012.02.25 18:32:27 2011.06.01 01:00 xEAx EURUSD,M15: File initx.csv read successfully
    2012.02.25 18:32:27 2011.06.01 01:00 xEAx EURUSD,M15: Testing mode detected.
    2012.02.25 18:32:27 2011.06.01 01:00 xEAx EURUSD,M15: Account Name: Aletheia
    2012.02.25 18:32:27 2011.06.01 01:00 xEAx v4.0 EURUSD,M15: Account Number: 871092
    2012.02.25 18:32:27 2011.06.01 01:00 xEAx EURUSD,M15: Account Currency: USD
    2012.02.25 18:32:27 2011.06.01 01:00 xEAx EURUSD,M15: ———— xEAx has been started———–

    The EA is fully registered and its key entered and it reports no errors on the screen.

    The tick file is nearly a GB in size.

    I even tried editing the registry to 5 sec delay, but no help.

    Can you help?

    • #17 written by birt February 25, 2012 (2 months ago)

      It’s a problem with the EA. Try the MACD Sample EA and see if that works fine. If it does, set the starting date of your backtest a few days from the start of your data, looks like the EA needs more bars to start working.

      • #18 written by Aletheia February 25, 2012 (2 months ago)

        Thank you so much Birt. I am very appreciative of your fast and very helpful responses.

        My data runs from 1st June 2011 to 31st Jan 12. The EA worked when I set the start date to 1st July 2012.

        If I download data from say 1st April 2012, I now feel sure the EA will work between the dates 1st June to 31st Jan.

        In order to clear the buffers, so to speak, and replace the hst and fxt files with the new ones, in MT4, is it best to simply delete them or just overwrite them with the newly created files?

        I shall buy your Tick Data Data Suite very shortly.

        Thanks.

        • #19 written by birt February 25, 2012 (2 months ago)

          I suppose you mean 1st April 2011 and yeah, you are very likely correct. It probably needs at most a few days worth of bar data seeing that there are 96 M15 bars per day.

          Re: “clearing the buffers”, make sure you exit Metatrader 4 then either overwrite or delete first, it doesn’t really matter.

          • #20 written by Aletheia February 25, 2012 (2 months ago)

            Yep, 1 April 2011, of course!

            Thanks for your help.

  • #21 written by lars March 3, 2012 (2 months ago)

    Hi,
    does the tick way (above) use with the MT5 ?

    • #22 written by birt March 4, 2012 (2 months ago)

      You cannot use any of the tick data tools on eareview.net with Metatrader 5. Due to several factors (including the complete failure of MT5 to gain popularity and the fact that backtesting works completely different there), I currently have no plans to make MT5 work with tick data.

  • #23 written by jerome March 8, 2012 (2 months ago)

    what’s the best CPU type to do back tests?
    MT4 is not multithread, like MT5 is, and doing backtest using tick data is slow.

    so I’m looking for best setup to improve the performance.

    thanks.

    • #24 written by birt March 8, 2012 (2 months ago)

      The best CPU for backtesting is that with the highest speed per core, which is typically that with the highest clock per core.

      That being said, you can run multiple MT4 backtests at once to get more from a machine.

  • #25 written by Harald March 8, 2012 (2 months ago)

    Hello,
    Sorry for the simple question.
    I created a demo account with Dukascopy (JForex). The Historical Data Manager gives the option Bid/Ask. Which one I should use for the download?

    • #26 written by birt March 8, 2012 (2 months ago)

      As specified in the detailed guide, you have to select Ticks in the Data Type field. When you do that, the option to select bid/ask will no longer be available.

  • #27 written by balgad March 11, 2012 (2 months ago)

    Hi Birt,

    first of all, your TDS trial is great and it worked flawlessly at first time. But next time I tried it (after downloading data in a more wild date range from Dukascopy) “zero divide” message can be seen in journal section of the MT4 terminal and backtesting stops at the first step of the process. Do you have any idea how to fix this?

    Thanks

    • #28 written by birt March 11, 2012 (2 months ago)

      That has nothing to do with the Tick Data Suite. It’s probably a bug in the EA you are testing and you can verify this by backtesting the MACD Sample.

      Since the issue is probably related to the lack of history data, I suggest attempting to start the backtest a couple of weeks after the first tick in your FXT (e.g. if you created your FXT starting with April 2007, set May 2007 as the start date for your backtest).

      • #29 written by balgad March 11, 2012 (2 months ago)

        Thanks, you were right, it was a bug in the EA, testing with MACD works. But after a successful run I see that the modelling quality is just 90% instead of 99%. Have I made a mistake during the merging and loading process? I used your PHP scripts (which skips downloading those data files that are already have been downloaded) and csv2fxt script but after inserting them into MT4 terminal it gave only 90% modelling quality. I repeated the whole process (with downloading again the base tick data) but sadly it remained the same.

        • #30 written by birt March 11, 2012 (2 months ago)

          You must be doing something wrong. Please take a look at the FAQ & Troubleshooting page (section 2, question 2, 3 or 4, can’t remember).

  • #31 written by Keith April 7, 2012 (1 month ago)

    Hi Birt,

    Exactly how do you run multiple instance of MT4 from the same folder?

    Thanks in advance.

    keith

    • #32 written by birt April 7, 2012 (1 month ago)

      If you’re using the Tick Data Suite, you just run tds.exe twice.

      Alternatively, you can use one of the loaders available on the downloads page but those only work up to MT4 build 409.

      • #33 written by Keith April 9, 2012 (1 month ago)

        I’m using Tick Data Suite yet when I open it twice (or more) it just reloads the same MT4 platform. I have checked thoroughly but only one instance is running.

        I’m using windows 7 and everything else sems to be working properly. Any ideas?

        • #34 written by birt April 9, 2012 (1 month ago)

          Are you by chance using Tick Data Suite version 1.0.6 with build 419? In that case, you need to upgrade to v1.0.7 which was released last week to fix precisely this issue. You can check your version by inspecting the Details section of the Properties window of tds.exe. Also, I recommend subscribing to the updates mailing list at http://eareview.net/tick-data-suite to easily keep up with the new versions.

          • #35 written by Keith April 9, 2012 (1 month ago)

            You got it in one:) Thanks.

            Downloaded 1.07 and it works fine.

  • #36 written by raindog April 9, 2012 (1 month ago)

    Hi Birt,

    I downloaded and installed TDS. Great program, which worked really well at first, but after converting my second CSV file and moving the files into the relevant folders, I keep getting the error “The application key is not valid. The patch was not applied.” Any ideas?

    • #37 written by birt April 9, 2012 (1 month ago)

      I reset your trial and extended it by a week. I will contact you on email for more information that could help me identify the issue and with some details that will help you get rid of it.

  • #38 written by raindog April 9, 2012 (1 month ago)

    Birt,

    As stated via e-mail, now when I run TDS I get the error “you cannot transfer your trial licence to a different computer.”

    • #39 written by birt April 9, 2012 (1 month ago)

      That was my bad – forgot to click the reset button after extending your trial.

  • #40 written by raindog April 9, 2012 (1 month ago)

    cheers, all fixed now!

  • #41 written by Harrison April 20, 2012 (3 weeks ago)

    Hello Birt

    Great site with some great tools. I have a query. Is it possible to use the csv2fxt script to generate custom HST files? I want to create historical range bars (http://www.investopedia.com/articles/trading/10/range-bar-charts-different-view.asp#axzz1sauJLJEl)

    All the scripts I’ve seen from other people use 1-minute bar data, and they create offline charts which are updated real-time (but it doesn’t quite work)

    Thanks

    • #42 written by birt April 20, 2012 (3 weeks ago)

      It’s not possible but the source code is downloadable so if you have a background in programming you could modify it to spit out range bars.

  • #43 written by Harrison April 23, 2012 (3 weeks ago)

    Yes, I thought about that, and I do have a background in programming and I have looked at the code, but what can I say? You are obviously a greater programmer than I. I will have a try though.

  • #44 written by Sam Adams May 11, 2012 (5 days ago)

    Hello, I am trying like crazy to get a 99% quality but am still only getting 90% any help would be great. I am downloading tick data its been converted, i’m clearing my history and downloading 1m for said pair, i am using a 15min for my test so the screen and data downloaded will run off the 15min. Any help would be awesome!!!

    • #45 written by jerome May 11, 2012 (5 days ago)

      you have to download the tick data using the birt’s tools.
      if you download from MT4 you’ll get 90% of quality only.

      follows the birt’S documentation to know how to create the tick data for your tests.
      generally you have to create the tick data for each timeframe you have to test, not only the 1min.
      so if your EA is tested on the 15min timeframe, you have to create the history data for this timeframe, and put the resulting file in the tester/history folder.

  • #46 written by Sam Adams May 11, 2012 (5 days ago)

    I also read that you can only achieve 90% quality if you download the M1 data, is this true?

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top