The tick data section of eareview.net is a detailed guide that will lead you through the whole process of tick data backtesting, starting from where to acquire free historical Forex tick data, how to download it and how to use it in backtesting Metatrader 4 expert advisors to obtain a 99% modeling quality. If you’re not sure what backtesting is, it’s probably a good idea to purchase the Metatrader Backtesting and Optimization Course, which is geared toward people that are new to Forex and Metatrader 4 backtesting.

This page is divided into several sections:

Scope

In general, backtesting using the data from the MT4 history center might be good enough for EAs that are not scalping or pip hunting. However, if you’re dealing with such an EA or any kind of EA that closes trades within 1-15 pips, even the smallest feed differences might have a very large impact.

The issue is caused by the Metatrader terminal not having access to the real tick data, but only to minute bar data in the best case, which forces it to give your strategy backtest “false” ticks generated through a process of interpolation using the data for the smallest timeframe available. This is most likely not important to an expert advisor that uses stoploss and takeprofit targets of over 100 pips, but in the case of robots that attempt to scalp a few pips here and there, your backtest could be completely misleading.

So, it is very important to try testing using data having a quality that is as high as possible which is why I put together some resources, all of which I use in my backtesting when needed.

Tick data guides

  1. How to download free tick data – details the download process using several free tick data sources: Dukascopy, Oanda, Pepperstone, Integral, MB Trading and Gain Capital.
  2. How to prepare your tick data for Metatrader 4 – guide to converting the tick data to a format compatible with Metatrader 4 (from CSV to FXT).
  3. How to backtest using tick data with Metatrader 4 – a review of the options available for using tick data with the Metatrader 4 platform.
  4. FAQ & Troubleshooting
  5. Downloads

The Metatrader Backtesting and Optimization Course

Using tick data requires a fair bit of computer proficiency. If you need more detailed instructions or if you run into issues not dealt with in the comments section, it might be an idea to head to the Metatrader Backtesting and Optimization Course (formerly named Guide to Getting Rich with Forex Robots). The guys there have put together some very nice video material detailing 99% backtesting step by step, designed to help the users who are not very experienced when it comes to computers, together with a FAQ with the most common problems you may encounter and with screenshots for each step in the video. However, their scripts are inspired from my scripts from the end of 2009, meaning that they are somewhat outdated now.

I’ll be honest with you: at first, when I saw the old name of the product (Guide to Getting Rich with Forex Robots) I thought it would be some bullshit, but then I had to check it out (due to the 99% tools used, part of which are originating from me) and I found it so useful for the less seasoned people that I ended up becoming an affiliate. In addition to the 99% backtesting info, they put together a lot of useful info about EAs, starting from the basics of backtesting, going in detail into money management strategies and also sporting a very thick “how to spot scam EAs” section that will give you a solid idea of what’s happening with many commercial EAs and what are the processes behind them. As a bonus, there’s a section dedicated to becoming a signal provider, but I’m not sure that’s the best way to go. In a nutshell, if you’re rather new to Forex or if you want a peek behind the curtains of selling EAs, this guide will probably save you a ton of money. They used to have a very good support system (I’ve had to use it myself) but nowadays I’ve heard several people complain about large delays in replying. Other than the support seemingly lacking as of late, the only major flaw I can find is its name: I would’ve rather called it “Guide to avoid Losing Tons of Money with Forex Robots”; these aside, it’s well worth the $87.

The main differences between their tools suite and what you can find here are:

  • If you choose to use the Birt’s patch script or Tick Data Suite, you do not need a loader unless you want to load multiple MT4 instances from the same folder.
  • In most cases, the 2GB FXT limit no longer applies.*

* The 2GB limit still applies to MT4 builds 402 or lower when using the Birt’s patch script and an older OS (e.g. Windows XP, Windows Server 2003).

Long story short, the Metatrader Backtesting and Optimization Course contains some info that complements that which you can find in my pages.

The Walk Forward Analyzer

Not directly related to tick data but with built-in support for it, the Walk Forward Analyzer is an excellent tool that lets you optimize your Metatrader 4 expert advisors in steps, in a technique called Walk Forward Analysis which is also described in the Metatrader Backtesting and Optimization Course. Simply put, you optimize your EA for say 3 months, then you test it for the next 1 month to see if the best parameters resulting from the optimization work fine on out-of-sample data, then you optimize it further on the next 3 months and so forth. This tool lets you automate the whole process and does all the runs for you, providing an exhaustive set of configuration parameters and a neat optimization report at the end. Priced at $29.95, it’s a veritable steal; the Walk Forward Analyzer is definitely a must-have for anyone doing serious EA development. But don’t take my word for it, visit the website and download the 14 day trial version. To top it all, Andrew – the programmer – is also very prompt with tech support as well as feature requests that make sense.

History

Since there are frequent updates to the tick data tools, I decided to keep a Tick data changelog that lists all the changes that the scripts have gone through.

Also, for those interested, the old tick data page is still available but a lot of the information there is now obsolete.

  • #2142 written by rajiv July 17, 2012 (10 months ago)

    I just check my EA with ducks data with real spreads always profits ,but from truefx data always losses with even 0 spread and no commission. for same EA HUH!
    I am 100% MAD

    • #2143 written by birt July 18, 2012 (10 months ago)

      You should add some debug information printing to the EA, test it in visual mode and find out why would that happen.

  • #2144 written by TonyBahama August 1, 2012 (9 months ago)

    Hi!
    I know this site is for Metatrader users, but I would like to try ForexTester as well, which is a bit more professional.
    Do you know any methods to import/convert Dukascopy real tick data to Forex Tester, or do you have any advice?

    Thanks very much!

    • #2145 written by birt August 2, 2012 (9 months ago)

      No, sorry. ForexTester is a paid product so I would suggest asking ForexTester support.

  • #2146 written by Dewey August 9, 2012 (9 months ago)

    Wow–this is just what I am looking for. I just converted a manual strategy I have been using into an EA and need to run all sorts of tests. I am going to get the Walk Forward Analyzer to start.

  • #2147 written by Dmitriy August 15, 2012 (9 months ago)

    Hi!
    I have a Renko EA. I can already save Renko as M5. Can I use TDS for testing Renko?

    Thank you!

    • #2148 written by birt August 15, 2012 (9 months ago)

      No, sorry, you can’t.

      • #2149 written by Vivian Hsu September 7, 2012 (8 months ago)

        Hi Birt,

        TDS is great but I’ve been working on Renko and just missing this piece (tick backtesting on Renko) before I have confidence to trade live. Any chance TDS will be enhanced later to support this? Many thanks!

        Regards,

        Vivian

        • #2150 written by birt September 12, 2012 (8 months ago)

          Don’t think so, sorry. Renko bars only go one way while tick data goes both ways so there’s no way to accurately represent it using Renko bars. The only possible representations would be inaccurate because the ticks outside the bar would likely have to be discarded.

        • #2151 written by Oblivian September 13, 2012 (8 months ago)

          I actually developing one expert, base on Renko charts. After two years of looking for solutions, and got on of the best programmer to study any available Renko bactesting script or Renko experts, 90% of the data are lost for ever. However, if you can still use those script or Renko experts to develop you system you will get a ”simulation”. You will never be able to compare backtest results with reality, because too much data are missing. Even more if you try to scalp one brick value. To be as close to reality with Renko you need to respect one simple rule. Your Stoploss and take profit, need to be at least 3 time the value of one brick. Just Google ”Renko plug-in backtest”. You should find something for backtest. That will not use your FXT to create chart history and the end result will be about 24,99% modelling quality. But remember that your strategy could be also good in live trading even if you can not prove it! Because I made it!

          • #2152 written by Vivian Hsu September 14, 2012 (8 months ago)

            Hi Oblivian, thanks for your reply. What do you mean by “90% of the data are lost for ever”? I manage to download every tick from Dukascopy for the past years.

            • #2153 written by birt September 14, 2012 (8 months ago)

              He specified that the data is lost only in case of Renko bars. It’s probably not 90% but a huge chunk of ticks happens outside the Renko bars.

              There are solutions for MT4 Renko backtesting but keep in mind that MT4 only interpolates ticks *within* the bar. In a live environment you will also have ticks outside the bar and most of the time the bar will head both ways before being completely formed.

  • #2154 written by Nick August 28, 2012 (8 months ago)

    Hi Birt,

    I’ve been using Tick Data Suite just fine with Dukascopy data for the past few months on Pepperstone’s platform. However, I’ve started testing with Armada Markets which has a .arm extension on their currency names. Despite adding this to the history files and the tester files, I cannot get a working backtest.

    Please advise.

    Kind regards,

    • #2155 written by birt August 29, 2012 (8 months ago)

      The FXT also has the currency pair name inside the file. If it doesn’t match, your backtest won’t start. You can edit that with a hex editor, it’s at offset 0xC4. If you still have problems after that, please use the support form and send me the error message you are receiving.

      • #2156 written by FxGuy August 29, 2012 (8 months ago)

        when you say “is at offset 0xC4″ which hex editor you use for the same, for editing that FXT file?
        Can it be any?

        • #2157 written by birt August 29, 2012 (8 months ago)

          Yes, any hex editor. Personally, I use hiew but I would definitely not recommend that to the average user. Something like WinHex or Neo Hex Editor should be better suited.

  • #2158 written by Nick August 29, 2012 (8 months ago)

    My aim is to have lotsizes greater than 50 in 99% back testing. It seems like regardless of the broker I use (Alpari UK, Armada, Pepperstone, FXPig), the lotsizes in back testing are capped at 50, whereas if I run the same test without a pre-made FXT there doesn’t seem to be an issue in reaching hundreds of lots. How do I fix this?

    Kind regards,

    • #2159 written by birt August 29, 2012 (8 months ago)

      The maximum lot size is taken from the broker that you were using when you created the FXT. If it’s 50 in your backtest, it means it’s 50 for that broker. Running a regular backtest with that broker will also cap the lot size at 50.

      There are two ways to fix this:

      • Recreate the FXT using a broker that has a higher max lot size.
      • Use a hex editor (see the FAQ for some more info) and edit the max lot value at offset 0×114. Keep in mind that it’s little endian so for a max lot of e.g. 100000 (which is 0x186A0 in hex) you’d have to fill in the bytes with A0860100.
      • #2160 written by Nick August 30, 2012 (8 months ago)

        Thanks Birt! Works a treat.

  • #2161 written by James September 17, 2012 (8 months ago)

    Birt, you are a life saver. The time your writeups have saved are tremendous, thank you.

    I have one question though, I run the optimizer and come up with what looks like a winning set of inputs. I then right mouse click and select set inputs and run the tester. The problem is I get completely different results.

    Have I missed something?

    • #2162 written by birt September 18, 2012 (8 months ago)

      It’s a bug in MT4. If you are specifying a start/end date for your optimization, only the first pass takes it into consideration. Subsequent passes use the whole range of the FXT. Try removing the start/end dates, it should make the output identical.

  • #2163 written by James September 18, 2012 (8 months ago)

    Thanks Birt. Works better without the dates.

  • #2164 written by Andy September 19, 2012 (8 months ago)

    Hi Birt, if I use several TF for optimization and backtesting, I can get them in one go with the script several boxes marked “Create MX”, or I have to use the script once for each TF?
    Thanks.

    • #2165 written by birt September 19, 2012 (8 months ago)

      You can create them in a single run using the Create* parameters.

  • #2166 written by andy November 19, 2012 (6 months ago)

    Hello Birt.

    First of all, thank you for all your info, is very helpful.
    I have two questionsL:

    2. I am testing a EA I wrote, the modelling quality is in 25% should I need to have that % in 99% in order to be more accurate? if so then do I need the TICK DATA? I f I dowload the TICK DATA with DUKASKOPY but I trade with another broker is the same??

    • #2167 written by birt February 27, 2013 (2 months ago)

      I seem to have missed your comment, sorry about the answering delay. Anyway, I’ll try to answer your questions in order.

      If your modeling quality is 25%, you are probably testing on the M1 timeframe; you would need tick data (and thus 99% modeling quality) especially if the EA you wrote can be described as a scalper or as a fast EA (closing trades within a few minutes). If your EA has an average of like 50-100 pips per trade, you will probably get very similar results with or without tick data.

      I don’t really understand your question about downloading tick data and trading with another broker – I will assume you are asking whether the Dukascopy tick data is the same as tick data from another broker and the answer is definitely not: the Dukascopy tick data does not coincide with tick data from another broker. However, how much of a difference that makes in a backtest depends entirely on the EA that is being backtested – for some it will be almost the same, for others it might yield completely different results.

  • #2168 written by ramin December 14, 2012 (5 months ago)

    Hi Birt,

    Is tick data suite take into account swaps in its backtesting?
    Regards

    • #2169 written by birt December 17, 2012 (5 months ago)

      When converting the tick data into an FXT, the current swaps from your broker are used. That is to say that yes, the swaps are taken into account but they’re the current swaps for the whole duration of the backtest. Unfortunately, there’s no historic swap data that I know of and even if there was, it would be impossible to use it with MT4 and it would also be different for each broker.

  • #2170 written by Marcel February 27, 2013 (2 months ago)

    Hi Birt,

    I have a question which is most likely already raised a couple of times but what I like to know if it’s possible to read the data in the header of the FXT file?
    And is it possible to check whether I am using fixed spread or real spreads in the FXT file?

    Regards,
    Marcel

    • #2171 written by birt February 27, 2013 (2 months ago)

      Certainly – it’s possible to read the data in the header but I don’t know if there’s any tool that does that and displays the info stored. The closest thing you could do is create an EA that reads the stored parameters and prints them in the log or on the chart and run that EA in a backtest. You can do the same for the spread, just print it in a Comment() on the chart and run the backtest in visual mode – if it fluctuates, it’s real spread.

      • #2172 written by Marcel February 27, 2013 (2 months ago)

        Thanks for your quick response Birt, yes that’s a possibility, I will take a closer look at it at a later stage. Thanks.

  • #2173 written by Andy March 11, 2013 (2 months ago)

    Hi Birt, wanted to know for MT4 using HST files and whether to write-protect?

    • #2174 written by birt March 11, 2013 (2 months ago)

      If you’re asking whether you should use the generated HST files, the answer is yes. As for write-protecting, only do that if your HST files also include recent data (less than a month old) and if you do not use that MT4 terminal for anything other than backtesting.

  • #2175 written by Nduru March 23, 2013 (1 month ago)

    hi birt,

    this is an unrelated question to hist data, Id like to know, my e.a does not place trades on a demo or live account because it needs that I first open the orders without stoploss or take profit then I modify them. is there a way I can avoid having to modify orders?

    • #2176 written by birt March 23, 2013 (1 month ago)

      Sorry, not really. I’m sure there are still some brokers who support the old way (SL & TP with the order) but I wouldn’t recommend trying them.

  • #2177 written by Marcel April 1, 2013 (1 month ago)

    Hi Birt,

    I am using the Tick Data Suite and downloaded and converted a lot of tick data and all went fine. Now I have installed a new MT4 terminal of another broker and went to the whole process again (installed the TDS.exe file, converted the csv-files into fxt- and hst-files and moved them to the right directories) but if I want to run a backtest of i.e 2010 up till now, it’s showing me the message that there is no data, but if I start the backtest at the beginning of 1984 is running smoothly, showing trade dates in 1984. So I guess something went wrong with the date conversion/processing. Do you know how I can solve this issue?

    Kind Regards,
    Marcel

  • #2178 written by P.A April 2, 2013 (1 month ago)

    HI
    I am backtesting within tds but my renko show 0 volume for each bar.. I downloaded the data from dukascopy.
    Also icustom renko to refer to a second renko chart in backtest gives no trade at all, Any idea?
    thanks

    • #2179 written by birt May 12, 2013 (1 week ago)

      Sorry, I did not develop or use anything renko-related so I’m afraid I can’t offer any assistance. I would advise asking the vendor of the renko package you are using. In any case, if there were no renko bars, 0 volume usually occurs when you are using real spread – due to the way the spread is encoded in the FXT, the volume information is lost.

  • #2180 written by Chris P April 18, 2013 (1 month ago)

    Just some input I wanted to add regarding the MetaTrader Back Testing & Optimization Course offered at http://www.guidetogettingrichwithforexrobots.com. Be wary that it is pretty outdated and actually pretty slim on content for what they charge. Also, and this is a biggie, they DO NOT HONOR their 60-day guarantee. I purchased the course and after going through it didn’t really learn anything new that I hadn’t already learned here on Birt’s site or figured out myself through trial and error. I both submitted a support ticket and emailed them (multiple times actually) requesting a refund, well before my 60 days were up, and they still have not refunded me or even replied/answered my ticket/email or anything! This has been weeks later now and still no word from them. I don’t know if they were just conveniently waiting until after 60 days or if they have just totally abandoned the product or what? Just wanted people to know, based on my experience here I cannot recommend their product and can verify that they DO NOT honor their guarantee stated on the website.

  • #2181 written by ramin April 22, 2013 (4 weeks ago)

    Dear Birt,

    I have a question maybe not relating to this topic but related to tick data,
    Is there any way to capture online tick data into a buffer or file in MT4?

    I know that I can write some indicator to get the quotes and write them on file, but here is a problem, it is possible that I miss some quotes, I want to capture the exact data that MT4 shows on its tick chart

    Thanks and Regards
    Ramin

  • #2182 written by SimonG April 22, 2013 (4 weeks ago)

    You will miss some ticks . . . at least I am always missing some ticks.

    If you want to know if you miss a tck keep a running count of the ticks for the current bar and compare it to Volume[0], if the numbers differ then you have missed a tick or more than one.

    • #2183 written by birt April 22, 2013 (4 weeks ago)

      Simon is correct although with some optimization and if you hide all other symbols and run a single chart with your logger you have very good chances of “catching” most ticks.

      • #2184 written by ramin April 23, 2013 (3 weeks ago)

        you are right, but I am wondered while mt4
        catch all of the ticks of course in a buffer that is transferred through TCP/IP packets why it does not make it possible to the programmer to access the those ticks in a buffer, thanks

        • #2185 written by birt April 23, 2013 (3 weeks ago)

          Because if that would happen for an EA that takes a long time to process a tick, it would queue up ticks and eventually reach a situation where the EA is processing ticks that occurred quite a while ago. What MT4 does is skip feeding each tick that comes while the EA is busy processing another tick.

          • #2186 written by ramin April 24, 2013 (3 weeks ago)

            Thats right, thanks

  • #2187 written by janggo May 11, 2013 (1 week ago)

    Hi Birt, do you have tick data for GMT +0 Broker such Exness ?

    • #2188 written by birt May 11, 2013 (1 week ago)

      I don’t really have any tick data of my own, but the Dukascopy data is GMT so you could probably use that. Anyway, the CSV2FXT script automatically aligns the data to GMT+0 and no DST for all supported tick data sources (unless you change its parameters, of course).

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top