The tick data section of eareview.net is a detailed guide that will lead you through the whole process of tick data backtesting, starting from where to acquire free historical Forex tick data, how to download it and how to use it in backtesting Metatrader 4 expert advisors to obtain a 99% modeling quality. If you’re not sure what backtesting is, it’s probably a good idea to purchase the Metatrader Backtesting and Optimization Course, which is geared toward people that are new to Forex and Metatrader 4 backtesting.

This page is divided into several sections:

Scope

In general, backtesting using the data from the MT4 history center might be good enough for EAs that are not scalping or pip hunting. However, if you’re dealing with such an EA or any kind of EA that closes trades within 1-15 pips, even the smallest feed differences might have a very large impact.

The issue is caused by the Metatrader terminal not having access to the real tick data, but only to minute bar data in the best case, which forces it to give your strategy backtest “false” ticks generated through a process of interpolation using the data for the smallest timeframe available. This is most likely not important to an expert advisor that uses stoploss and takeprofit targets of over 100 pips, but in the case of robots that attempt to scalp a few pips here and there, your backtest could be completely misleading.

So, it is very important to try testing using data having a quality that is as high as possible which is why I put together some resources, all of which I use in my backtesting when needed.

Tick data guides

  1. How to download free tick data – details the download process using several free tick data sources: Dukascopy, Oanda, Pepperstone, Integral, MB Trading and Gain Capital.
  2. How to prepare your tick data for Metatrader 4 – guide to converting the tick data to a format compatible with Metatrader 4 (from CSV to FXT).
  3. How to backtest using tick data with Metatrader 4 – a review of the options available for using tick data with the Metatrader 4 platform.
  4. FAQ & Troubleshooting
  5. Downloads

The Walk Forward Analyzer

Not directly related to tick data but with built-in support for it, the Walk Forward Analyzer is an excellent tool that lets you optimize your Metatrader 4 expert advisors in steps, in a technique called Walk Forward Analysis. Simply put, you optimize your EA for say 3 months, then you test it for the next 1 month to see if the best parameters resulting from the optimization work fine on out-of-sample data, then you optimize it further on the next 3 months and so forth. This tool lets you automate the whole process and does all the runs for you, providing an exhaustive set of configuration parameters and a neat optimization report at the end. The Walk Forward Analyzer is a must-have for anyone doing serious EA development. But don’t take my word for it, visit the website and download your copy – the WFA used to be priced around $30 but recently the author decided to provide it for free.

History

Since there are frequent updates to the tick data tools, I decided to keep a Tick data changelog that lists all the changes that the scripts have gone through.

Also, for those interested, the old tick data page is still available but a lot of the information there is now obsolete.

  • #2019 written by rob February 27, 2012 (4 years ago)

    I have looked at csv file etc, but cannot see anything that would give a rise to a difference. Also I have tried running yr script on a different EA and including all ticks by inserting “break;” command before comparison line and this gives a different result again. I can only assume MT4 ignores some ticks if they come in rapid succession, as is the case if running real time.
    rgds

    • #2020 written by birt February 27, 2012 (4 years ago)

      MT4 does not ignore any ticks in backtesting if they are in the FXT file. Feel free to make an EA that counts the ticks and compare with the number received at the end of the FXT creation.

      I’m afraid I lack the time to help you any further in your quest to find the differences. You are using a CSV2FXT copy that you are modifying on your own and I simply don’t have the time to look into what possible consequences your changes could have.

      Finally, what I can tell you is that you have to change the EA code and record (log) all the values that could possibly be different (indicators, timestamps etc.) at the critical points (when the problematic orders are opened or closed) to find out what causes the discrepancies.

  • #2021 written by Monica February 28, 2012 (4 years ago)

    Dear Birt,

    I also forward tested and back tested a simple MA cross (tradebar=1) and got different results. I downloaded tick data from Dukascopy, made both “hst” and “fxt” files read only and tested both with and without “RealSpread”. I forward tested using a real account with FXOpen (I’ve heard they get data from Dukascopy).
    Could you please tell me if you get same results on back testing as on forward testing? And what broker do you use, what tradebar (shift) value do you use? Any recommendations are most welcomed.

    Thank you,
    Monica

    • #2022 written by birt February 28, 2012 (4 years ago)

      It’s not unusual to get different results in backtesting when comparing to forward testing. In fact, it’s quite common. Backtesting is meant to give an rough indication of what the EA is capable of; it happens quite rarely to be able to fully reproduce live results in backtesting. As a guideline, the more indicators an EA is using, the larger the differences will be.

      On top of that, to the best of my knowledge, FXOpen no longer uses Dukascopy for their price feed for some time. Are you using a timeframe larger than H1? If so, have you configured the GMT offset correctly?

  • #2023 written by UKMike March 2, 2012 (4 years ago)

    Birt,

    I have just had a go at using the new PHP process code to cope with the bi5 files. I have put the 7za command line app in the php folder, but I’m getting errors like this:

    Warning: 0 sized eurusd/2012/01/26/21h_ticks.bi5
    eurusd/2012/01/26/22h_ticks.bi5
    PHP Warning: file_get_contents(C:\Users\Mike\AppData\Local\): failed to open stream: No such file or directory in C:\php\process_dukascopy_data.php on line 206

    Warning: file_get_contents(C:\Users\Mike\AppData\Local\): failed to open stream:
    No such file or directory in C:\php\process_dukascopy_data.php on line 206
    PHP Warning: unlink(C:\Users\Mike\AppData\Local\): Permission denied in C:\php\process_dukascopy_data.php on line 207

    Warning: unlink(C:\Users\Mike\AppData\Local\): Permission denied in C:\php\process_dukascopy_data.php on line 207
    Error: unable to read extracted file

    Initially I thought there were files set as read only that it needed to change, or that 7za couldn’t run becuase Win7 had decided to block it as it was downloaded, but I don’t think it’s either of those things now.

    Any ideas, please?

    Thanks, Mike

    • #2024 written by Alex March 4, 2012 (4 years ago)

      Birt,

      Thanks a a lot for updating the PHP scripts for the new format. Highly appreciated!
      I did try to run it with 7za.exe, version 9.20.

      I see the same error for all currency pairs:

      dukas_php2>php process_dukascopy_data.php GBPUSD 201201 201202 GBPUSD.csv
      Processing data starting from 01/01/12 00:00:00 up to 02/01/12 00:00:00 GBPUSD/2012/00/25/00h_ticks.bi5
      Error: failed to extract [GBPUSD/2012/00/25/00h_ticks.bi5]

      I run on clean Windows XP SP3 in a virtual machine on MacOS. As this is XP there is no all this security nonsense about virtual file systems and permissions.

      What could be the problem?

      • #2025 written by birt March 4, 2012 (4 years ago)

        There was a bug which was fixed in v0.22. Please download the new version and let me know if you still experience any problems.

        • #2026 written by UKMike March 4, 2012 (4 years ago)

          Hi,

          I am still getting errors using v0.22. Here are two different runs of the php process using different timescales, but both seemingly going wrong around the 25th/26th of the month, even though it’s looking at different months.

          C:\php>php process_dukascopy_data.php eurusd 201201 201202 test1.csv
          Processing data starting from 01/01/12 00:00:00 up to 02/01/12 00:00:00
          eurusd/2012/00/25/00h_ticks.bi5
          PHP Warning: file_get_contents(C:\Users\Mike\AppData\Local\): failed to open st
          ream: No such file or directory in C:\php\process_dukascopy_data.php on line 206

          Warning: file_get_contents(C:\Users\Mike\AppData\Local\): failed to open stream:
          No such file or directory in C:\php\process_dukascopy_data.php on line 206
          PHP Warning: unlink(C:\Users\Mike\AppData\Local\): Permission denied in C:\php\
          process_dukascopy_data.php on line 207

          Warning: unlink(C:\Users\Mike\AppData\Local\): Permission denied in C:\php\proce
          ss_dukascopy_data.php on line 207
          Error: unable to read extracted file

          C:\php>php process_dukascopy_data.php eurusd 201202 201203 test2.csv
          Processing data starting from 02/01/12 00:00:00 up to 03/01/12 00:00:00
          Warning: 0 sized eurusd/2012/01/25/00h_ticks.bi5
          (…and all the files in between these…).
          Warning: 0 sized eurusd/2012/01/26/21h_ticks.bi5
          eurusd/2012/01/26/22h_ticks.bi5
          PHP Warning: file_get_contents(C:\Users\Mike\AppData\Local\): failed to open st
          ream: No such file or directory in C:\php\process_dukascopy_data.php on line 206

          Warning: file_get_contents(C:\Users\Mike\AppData\Local\): failed to open stream:
          No such file or directory in C:\php\process_dukascopy_data.php on line 206
          PHP Warning: unlink(C:\Users\Mike\AppData\Local\): Permission denied in C:\php\
          process_dukascopy_data.php on line 207

          Warning: unlink(C:\Users\Mike\AppData\Local\): Permission denied in C:\php\proce
          ss_dukascopy_data.php on line 207
          Error: unable to read extracted file

          It’s not obvious to me why it’s reporting on 0 sized files in the second example when it didn’t say anything about the zero sized files in the previous weekend! The eurusd/2012/01/26/22h_ticks.bi5 file is the first non-empty one after that weekend.

          Sorry for the long post – hope the output helps to deduce what’s happening.

        • #2027 written by Alex March 4, 2012 (4 years ago)

          Birt,

          Yes, the 0.22 script still gives me errors. The problem is with spaces in 7za command line, for example:

          7za e -oC:\Documents and Settings\alex\Local Settings\Temp\tic9.tmp XAUUSD/2012/00/25/00h_ticks.bi5

          does not work, but if I manually change that to:

          7za e -o”C:\Documents and Settings\alex\Local Settings\Temp\tic9.tmp” XAUUSD/2012/00/25/00h_ticks.bi5

          then all is fine.

          • #2028 written by Alex March 4, 2012 (4 years ago)

            also, this line:

            $extracted = substr($extracted, 0, strrpos($fname, ‘.’) + 1);

            should be “-1”:

            $extracted = substr($extracted, 0, strrpos($fname, ‘.’) – 1);

            I have set my script to use c:\temp (plus the change above) and it works now!

            thanks again Birt!

            • #2029 written by birt March 4, 2012 (4 years ago)

              v0.23 should fix all the above problems. Thanks for pointing out the quote thing, Alex.

              • #2030 written by Alex March 4, 2012 (4 years ago)

                Birt,

                There are still issues with the PHP process script.
                First is the volume numbers. They seem to be random from one run to another. And they do not make sense anyway:

                2012.01.25 00:00:00.595,1.56235,1.5626,0,-491
                2012.01.25 00:00:01.143,1.56235,1.56258,0,-491
                2012.01.25 00:00:02.584,1.56231,1.56256,-852420752251127149690038958161920,0
                2012.01.25 00:00:02.914,1.56227,1.56252,-1014746758418535938326003712,936699903553806977145839288320

                Second issue is, for example for GBPUSD the first rows in the file for january are reported above. They start from Jan 25, not Jan 1. I think it has something to do with the loop:

                for($i = $starttime + 24 * 24 * 60 * 60; $i < $endtime; $i += 3600) {

                Can you have a look please?

                • #2031 written by UKMike March 4, 2012 (4 years ago)

                  I can confirm that the CSV files all seem to start on the 25th of the first month stated in the command line – hence my comment about there being no warning about zero length files in my previous post. I have tried this with three different start months, and seen the same problem. But the file errors have gone away, and the CSV file seems fine apart from the late start date. Thanks – nearly there, I think!

                • #2032 written by birt March 4, 2012 (4 years ago)

                  Damn, that was a lot of bugs in v0.21. I was in a hurry to leave (was away for a couple days) and kind of rushed it.

                  The CSV starting from the 25th of the month was something I added to be able to test if it works fine with both bin and bi5 files. In v0.24 it should start from the 1st of the month and the volume issue is also fixed now. Hopefully these were the last bugs.

                  • #2033 written by UKMike March 4, 2012 (4 years ago)

                    v0.24 looks just fine. Thank you!

              • #2034 written by Andy July 17, 2012 (4 years ago)

                I am still seeing the same error even with v0.25 which I downloaded yesterday. This error is stopping the entire process. Appreciate any fixup or workaround.


                AUDUSD/2008/07/25/01h_ticks.bi5
                Error: unable to read extracted file

  • #2035 written by Des March 3, 2012 (4 years ago)

    Hello Birt and all,

    First thanks in advance for your continuos help.
    I have updated my EURUSD historical data from time to time using JForex.
    Now, I have downloaded data till 01.03.2012, when doing the conversion and testing, I see now the minimum lot size allowed is 3 lots.
    Do you now how to fix this issue?

    Thanks again
    Des

    • #2036 written by birt March 4, 2012 (4 years ago)

      The minimum lot size allowed is dependent on your account. If the minimum on your account is 3 lots, that will also be the minimum in your FXT file. Take a look at the FAQ if you want to know how to modify it manually in the FXT or alternatively create the FXT using an account that allows a smaller minimum lot.

      • #2037 written by Des March 4, 2012 (4 years ago)

        Thanks for your answer, the strange thing is I have used this account many times with data converted from JForex, and this is the first time I have got this issue. By the way the platform has not been updated. Cheers

        • #2038 written by birt March 4, 2012 (4 years ago)

          Make a script containing Comment(MarketInfo(Symbol(), MODE_MINLOT)) and run it on a chart with the symbol you’re trying to use. It will probably say 3. If it doesn’t, I suggest recreating your FXT, it might’ve been a temporary configuration change by the broker (this can be changed on the server side without any platform update).

  • #2039 written by iDouble March 7, 2012 (4 years ago)

    Hello,

    Just the facts.

    – TrueFx data source (May 2009 through Jan 2012)
    – GBPUSD, EURUSD, USDJPY
    – Respective CSV files concatenated using “type” command at CMD
    – All CSV2FXT files installed appropriately
    – Ran CSV2FXT script from Navigator
    – Configured Script Window (FXT/GMT offset = 0 and FXT/DST = 2)
    – FXT & HST creation = M5 through MN (all true)
    – Clicked “OK” and script runs with % completion upper left chart window

    Experts Tab Log:

    1) 2012.03.06 17:49:38 CSV2FXT GBPUSD,M1: Alert: Possible error: large gap after 2010.02.08 23:59:57 (8.0 days).

    2) 2012.03.06 17:50:55 CSV2FXT GBPUSD,M1: Error in the CSV file: encountered older timestamp(s) right after the tick at 2010.03.03 23:59:00 (older timestamp: 2010.03.02 00:00:00).

    – Current FXT2HST completion reads: 72% (took approximately 20 mins)
    – FXT2HST process still running (but slowly for a box this fast)

    My Box:

    Brand New Dell Inspirion 620
    Core i3 Dual 3.30Ghz Processor
    6GB 1066Mhz DDR3SD RAM
    500GB 7200 RPM HD
    Intel H61 chipset
    FXCM MT4 V4/B409 (installed into separate directory for testing)

    Help?

    Thanks!

  • #2040 written by iDouble March 7, 2012 (4 years ago)

    Edit:

    BTW – I ran the concatenation from inside the directory containing the CSV files as outlined on Birt’s EA Review:

    c:\path to CSV files\type EURUSD-2009-05.csv EURUSD-2009-06.csv EURUSD-2009-07.csv, etc, etc, etc… EURUSD-2009-12.csv EURUSD-2010-01.csv EURUSD-2010-02.csv, etc., etc., etc… EURUSD-2010-12.csv EURUSD-2011-01.csv, etc., etc., etc., up through EURUSD-2012-01.csv. > EURUSD.csv

    The same type command and format was used for GBPUSD and USDJPY. All files were concatenated appropriately. So, unless TrueFX has an “8 day gap” in its data, I doubt the concatenation is cause for the two (2) anomalies seen above in my first post.

    Thanks.

    • #2041 written by birt March 7, 2012 (4 years ago)

      There’s a very simple way to check. Open the CSV file with a viewer or an editor that is able to open such large files (I use the viewer from Total Commander, it works great) and scroll down to the lines mentioned in the experts log. I’m pretty sure there’s no mistake by the CSV2FXT script and the 8 days are actually missing and the misplaced day is there as well.

    • #2042 written by birt March 7, 2012 (4 years ago)

      In your previous post, you mentioned the process is slow – actually, it’s pretty normal for a conversion to take 30-40 minutes. It’s a bit faster with SSD; having a fast CPU and lots of RAM doesn’t have a very large impact on it.

  • #2043 written by iDouble March 7, 2012 (4 years ago)

    Used Total Commander (nice) and viewed file to find this:

    GBP/USD,20100208 23:59:57.466,1.55711,1.55747
    GBP/USD,20100217 00:00:00.324,1.57825,1.57861

    The key here in found in the Date strings 20100208 and 20100217. 17-8 (accounting for the time differential) is approximately 8 days. So, it is confirmed to be a problem with the CSV and not CSV2FXT. Thanks for the tip and the help!

    I selected TrueFX, because your data selection guide indicated that TrueFX was Pro for having quality data. Fortunately, I have not run into more than one such gap with GBPUSD and can simply skip over the month of February for that particular year when back-testing.

    Can’t win them all, might be the lesson I learned in this process. However, using DownLoadThemAll (Fox Plug-in) made the choice of TrueFx even easier. 🙂 DLTA works like a charm and the concatenate command rounds it out as being one of the easier bulk loads for MT4 back-testing to get onto your HDD. So, my reason for going with TrueFX, was indeed a calculated decision.

    Going forward with FXT2HST on EURUSD and USDJPY, now that I know what to expect. Thanks again!

    [for those that use Firefox and do not have DLTA installed – you are missing a really good browser tool!]

  • #2044 written by Pirako March 8, 2012 (4 years ago)

    Hello all! Hello Bird,
    thanks a lot for providing these information- very helpful!

    My little problem:
    I am using DukasCopier 0.4 beta to get CSV for EURUSD and then apply CSV2FXT to use in MT4. So far so good!

    If i try to Backtest an EA which uses the “Bars” variable (for example to check whether there are more than 100 Bars on the chart) the Bot continuesly says “Bars less than 100”.. i did not have this problem with normal Backtesting using MT4 internal data.

    My EA is MegaDroid.

    Anyone has an idea how to fix this? Am i doing something wrong?

    Lots of pips to you,

    greetings

    • #2045 written by birt March 8, 2012 (4 years ago)

      Did you copy the history files? Do other EAs such as the MACD sample work fine on the same data?

      • #2046 written by Pirako March 9, 2012 (4 years ago)

        Hey Birt!
        Thanks for your answer, yes i can run MACD Sample with 99% Quality.. MD gives me:

        2012.03.09 12:49:52 2011.01.07 00:59 MegaDroid EURUSD,H1: Bars less than 100

        Any idea?

        • #2047 written by birt March 9, 2012 (4 years ago)

          Well, like I asked before – did you copy the history files in their designated location?

          • #2048 written by Pirako March 9, 2012 (4 years ago)

            Yes i followed the presented steps:

            http://img6.imagebanana.com/img/3f920qm8/Untitled.gif

            generated both history files at once, strange they have the same size!

            A clue?

            • #2049 written by birt March 9, 2012 (4 years ago)

              Try to start your backtest 7 days past the first date of your tick data file by configuring the start date in the backtest UI. You can set 1970.01.01 or any future date for the end date to let it use the file until the end.

            • #2050 written by Pirako March 9, 2012 (4 years ago)

              That indeed did the Trick! Started 7 days later and it works. I cant set the final date to the past tho.. doesnt work for me.

              Thanks

              • #2051 written by birt March 9, 2012 (4 years ago)

                You have to either set it to the future or otherwise exactly to 1970.01.01. Glad to hear you got it running.

                • #2052 written by socalengineer April 11, 2012 (4 years ago)

                  Hi Birt,

                  I am having similar issues to Pirako. I was getting the “Bars less than 100” error as well, but when I changed the start date that went away. My issue now is that Megadroid is giving 0 trades. The MACD example works perfectly on the same data. What do you think may be the problem?

                  I have copied the history files (.hst) and .fxt files to the appropriate places, double and triple checked.

                  • #2053 written by birt April 11, 2012 (4 years ago)

                    You probably created the files during the weekend when spread was wider and left the Spread parameter configured to its default value of 0. Your FXT file likely has a fixed large spread and Megadroid doesn’t take any trades because it is too large. If you want to verify this, just make a new EA and write in its start function:
                    Comment("Spread: " + DoubleToStr((Ask - Bid) / Point, 1));
                    and backtest it in visual mode.

                    Check out the FAQ if you feel like using a hex editor to modify it. Otherwise you can just recreate the FXT and pay attention to the spread parameters.

                    • #2054 written by socalengineer April 11, 2012 (4 years ago)

                      The spread was 2.0.

                      But good thing you pointed out visual mode, that helped me find out that Megadroid isn’t authenticating! Newb mistake. I have contacted the vendor to see why it is not authenticating on build 409. I will probably end up buying your Tick Data Suite

  • #2055 written by iDouble March 8, 2012 (4 years ago)

    Hello Birt,

    Just so you know, FXT2HST on EURUSD and USDJPY ran perfectly using TrueFX, with the exception of the exact same GAP and Time Overlap errors – on the exact same dates & times for each pair.

    Seeing that confirmed for me that TrueFX had a problem with their CSV compilation process internally. I have contacted them via email and and wrote a short analysis of the problem. I asked them for a fix. If/when they provide the fix, I will post the news here, so that others will know.
    —————————-

    I have one question for you. It involves the MT4 Strategy Tester’s Symbol Properties window. After you prepare your back-test and click on Symbol Properties, it displays a windows showing the “Spread” among other things. I noticed that the spread listed in that window for GBPUSD M1 was 16 pips. I set CSV2FXT to used a fixed spread at 3 pips.

    Starting the back-test, pausing and then calculating the Bid/Ask delta resulted in a 3 pip spread on the M1 chart. So, this confirms that CSV2FXT does override MT4’s Symbol Properties in that regard.

    I was skeptical at first given all the unworkable MT4 back-testing solutions out there, but after verifying each step of your process, I owe you a congratulations and a thanks! It works, if you follow the process.

    For now, I will simply skip the 8 day gaps in each pair and discount the single row M1 time overlap, which in the larger scheme of things is minor. Most of the other TrueFX data looks fairly good and having the ability to set my own spread, gives me additional testing parameters that I never had before.

    I can basically keep widening the spread, until my EA breaks – which in it self, is another form of Stress Testing the trade logic.

    Thanks again for the good work! 🙂

    • #2056 written by birt March 8, 2012 (4 years ago)

      Glad you got it sorted out!

      One problem with TrueFX data, though – if I recall correctly, if you use the whole available range, you will end up with an FXT file larger than 7GB. At the moment, even though the Tick Data Suite extends the MT4 range above 2GB, it is limited to 4GB due to the variable used to hold the file position being stored on 32 bits. This means that your backtest will stop somewhere past its half and completely disregard any data that follows. Although this is pretty darn complicated to work around, one of the next Tick Data Suite versions is going to allow using files over 4GB as well.

  • #2057 written by iDouble March 10, 2012 (4 years ago)

    Hello Birt,

    After spending hours with figuring out which historical data source to download, reading the EAReview site about back-testing, going through the steps meticulously to make certain that I was doing everything correctly, downloading huge amounts of data, going through all of the conversion steps and waiting patiently for files to be converted and copied; I get just one day of enjoyment from the back-testing.

    Why?

    Because, MT4 has just been “upgraded” to Build 216. Birt’s patch now suggests that I try something else to solve the MT4 back-testing problem because it will not run an any build beyond 409.

    I spent vastly more time getting this set-up correctly, than I did actually using it. LOL! That’s so funny! All that work, seemingly for nothing at this point.

    Do you plan to release a new “Birt’s Patch” that won’t be outdated by MT4 Build changes? That’s probably difficult to do, since the patch is what modifies the build, LOL. Hilarious!

    I’m not laughing at you – you did a great job. I’m just laughing at how long this took me, only to run two back-tests in total, before MT4 got a build change to 216. Absolutely, hilarious! All that time – all that patience and this is my reward. LOL! All you can do is laugh out loud, Birt.

    Again, not your fault – your stuff worked.

    For me. It will be back to the drawing board on MT4 back-testing!

    BTW – under Build 409, each time I ran a back-test using the Birt’s Patch successfully, and then clicked on the Back-Tester’s Terminal button (that tiny little X at the top left of the Terminal Window), it would Force Close MT4 each and every time. After re-starting MT4, opening the tester and clicking on the Close Terminal Window button, it only closed the Terminal Window and not the entire MT4 application. So, there was something amiss with that.

    But, it is all for not now, since BP v0.21 won’t work under MT4 build 216.

    What a lovely week this has been for me. 🙂

    • #2058 written by birt March 10, 2012 (4 years ago)

      The Birt’s patch script is now unsupported and is unlikely to get updated. Every such MT4 update requires a lot of work on my part (for instance, making TDS compatible with build 414 was over 24 hours of work), which is precisely why the Tick Data Suite is using a recurring billing model.

      If you wish to continue using the patch script, the solution – as you found out on your own – is to simply keep a MT4 build 409 installed and avoid updating it. You could simply delete liveupdate.exe from its folder to ensure it never offers to update.

      As of now, the Tick Data Suite is already compatible with build 416.

      • #2059 written by Mike March 13, 2012 (4 years ago)

        Does anyone have a copy of Hot Forex’s MT4 version 409? Or any broker for that matter? I already effed up and updated to 416 and don’t have the installer for 409

        • #2060 written by Murray March 13, 2012 (4 years ago)

          I can send you either Axitrader or SynergyFX install files for version 409 if that helps. Just get Birt to pass on your email address to me if you want them.

  • #2061 written by iDouble March 10, 2012 (4 years ago)

    Of course, I’m talking about Build 416, not 216. 409 was upgraded to 416, for clarification. 🙂

    I hope everyone has a great weekend! 🙂

  • #2062 written by iDouble March 10, 2012 (4 years ago)

    416 not 216 – correction.

  • #2063 written by iDouble March 10, 2012 (4 years ago)

    Well, Birt.

    Being a bit frustrated at the time spent with no glory, I decided to see if I could do a roll-back on 416. I rolled MT4 back to 409, into the same install directory that I always use for testing.

    C:\…..Live MT4\[this path will contain 416]
    C:\…..Test MT4\[this path will contain 409]

    I will let the Live MT4 path continue to upgrade its build, while never allowing the Test MT4 path to upgrade its build. Since the Test MT4 path will never be used for live trading, and the Live MT4 path will never be used for testing, I don’t see this as an immediate problem – and in fact, might be a longer-term homemade patch of sorts to the constant build upgrades that MetaQuotes is known for.

    It is not elegant, but it works. Birt’s Patch, now runs once again under the Test MT4 path.

    Unless, you can think of some really bizarre reason why I should not continue this practice with MT4 (keeping one path for Live Trading and another path for Back Testing, while freezing the Build on the Test path), then I will continue to use this method.

    Sorry, for the earlier drama, but I’m a full-time trader and I can’t afford to spend time that is unaccounted for. I got a little frustrated and blew off some steam. But, it looks things might work out after all – unless you see some glaring error in doing it this way.

    Thanks!
    iDouble

    • #2064 written by birt March 10, 2012 (4 years ago)

      Your setup should be working fine if you have no issue with the limitations of the Birt’s patch script. Currently, to the best of my knowledge there’s no reason to update MT4 further than build 409.

  • #2065 written by Neo March 10, 2012 (4 years ago)

    Hey iDouble,

    My heart goes out to you, because I run into this situation myself quite often. I prefer GoMarket’s MT4 for my back testing. I always use demo accounts for back testing. GoMarkets only give a 30 day demo (as do many brokers)…so at the end of the 30 days I have to renew my account(s) if I want to continue using it…and when there is an MT4 update, you’re forced to take it (like I was today). So all my FXT files in 3 separate GoMarkets folders are not working till there’s a TDS update that supports build 416.

    I use many other MT4s as well…that don’t limit a demo account to 30 days…but even they will act up at some point (even if you delete the liveupdate.exe file (so it doesn’t ask you to update each time you start MT4)…at one point I was not able to use the old 402 build, even though my demo account didn’t expire, the platform wouldn’t work properly, though I could still get into my demo account.

    All this to say that if it was possible to use an old MT4 build forever…I’d be doing that too. I think the main problem is connecting to your account, whether demo or live…at some point the broker can make it hard (maybe impossible) to login without being on the current build.

    Best Reagrds,
    Neo

    • #2066 written by birt March 10, 2012 (4 years ago)

      You’re not forced to take the update – you can still create demo accounts with MT4 build 414 as far as I know. Anyway, TDS v1.0.5 with support for build 416 is already uploaded, but just for reference it won’t happen that fast for every new build. Adding compatibility with MT4 builds can take much longer in some cases. As such, you’re advised not to update your backtesting MT4 installations unless you really have to or a compatible Tick Data Suite version is available. It’s safe to assume that the TDS will need an update every time a new MT4 build is released and this update will generally take 1-7 days.

  • #2067 written by iDouble March 10, 2012 (4 years ago)

    Hello Neo,

    Divisa Capital FX (DCFX) is a technology driven Forex intermediary. They house both retail and institutional Forex traders. At heart, they are Currenex shop, but they also provide MT4 with a Currenex data feed on their own server without a Boston Technologies bridge. Their MT4 demo accounts never expire.

    You don’t have to accept the MT4 live update. To turn off MT4 live updates, simply go to: c:\…….\[Your MT4 Install Directory]\. Find and delete LiveUpdate.exe.

    No more MT4 updates.
    —————————-

    Birt,

    Thanks. The dual MT4 path set-up with removal of LiveUpdate.exe seems to work fine.

    Tanks again.

    • #2068 written by Neo March 11, 2012 (4 years ago)

      Thanks for the heads up, iDouble. Haven’t heard of Davisa Capital before…just installed their MT4 (which coincidentally is still on Build 409) and opened a demo account. Looking good so far. I’ll give them a run and see if they do good for back testing. I like GoMarkets because they don’t have a limitation for how far market price must be to enter a pending order (OOpen price you set must differ from Market price by at least xx pips.”). Seems Davisa might not have that limitation either. Some brokers have a 9 pips requirement for some pairs which makes EAs that use pending orders pretty hard to work with.

      For disabling the auto-update feature, I normally just change the file name to something different, like “liveupdate-off.exe”…that way it still is in the folder in case you want to enable it later to take the update.

      Kind Regards,
      Neo

  • #2069 written by iDouble March 10, 2012 (4 years ago)

    Thanks again!

  • #2070 written by iDouble March 10, 2012 (4 years ago)

    I hope that was not a Freudian Omen’esk slip. :O

  • #2071 written by Oblivian March 11, 2012 (4 years ago)

    Hello Birt, I searching a way to use an MT4 backtest on M1 as a price feed stream server. I would like to connect another MT4 instance to that fake server. From the second instance running a ”fake” forward test.
    Do anybody made it? As a Renko and RangeBars trader or other special type of charting etc.. things would be much easier like that.
    Do any web server provide that type of fake price feed?
    If no one, I will open it, and get a great amount of members…..
    This is maybe a survey : )
    Do any body would like to have it?

    According to the first instance backtest speed, the forward test on instance two, would run at the same speed.
    Thank you!

    • #2072 written by birt March 11, 2012 (4 years ago)

      I didn’t really understand what exactly is it that you intend to do, but if you’re trying to make Metatrader 4 connect to a server of your own (aka not a Metatrader 4 server), you might find that more complicated than you can tackle unless you have very extensive development resources.

  • #2073 written by Oblivian March 12, 2012 (4 years ago)

    I want to make a fake broker server that use FXT file to generate price feed for any given time and pair. Maybe a PHP script could run an fxt file and provide price feed to mt4 like real-time price feed from a broker, except that I could speed up the market watch clock speed. Do any body can figure how that can be made?

    • #2074 written by birt March 12, 2012 (4 years ago)

      I’ll give you a short answer: you can’t.

      The MT4 communication protocol is proprietary (in other words, there’s no info on it anywhere) and unless you have someone who can reverse engineer it, you’re stuck at that point.

  • #2075 written by Joao Santos March 14, 2012 (4 years ago)

    Is the data scrolled back on a chart the most reliable to backtest? Since it is the unaltered broker data, I assume that I can backtest as far back as the scroll will allow me go. Then I restart the platform and backtest on the time frame that I did the back scrolling.

    Is this method reliable?

    • #2076 written by birt March 14, 2012 (4 years ago)

      The only issue is that you don’t get the actual ticks. You still get fake ticks when MT4 interpolates them.

  • #2077 written by Jim March 20, 2012 (4 years ago)

    Is it possible to test multi-timeframe EAs with dukas tick data?

    Thanks

    • #2078 written by birt March 20, 2012 (4 years ago)

      Certainly. However, bear in mind that you must also put the HST files in the proper location (as specified by the guide) because the indicators are using the data from the HST files.

      • #2079 written by Jim March 22, 2012 (4 years ago)

        Do I have to attach birt’s patch to both charts? My EA uses H4 and M30

        • #2080 written by birt March 22, 2012 (4 years ago)

          No, you don’t. Please note that no support is provided for the birt’s patch script. However, you should be ok if you follow the guides and read the FAQ if you run into any problem.

  • #2081 written by iDouble April 5, 2012 (4 years ago)

    Hello Birt,

    I’ve been running your script for a few weeks now and I’ve noticed two things:

    1) The Input Tab always maintains the configuration from the last EA that was run. That might sound like its normal, but when you make a change in your EA, the Inputs Tab of the Tester never changes. So, unless you remember each change made, the EA that runs is not running with the parameters you coded into your EA.

    2) Each time you run the script, and then run a back-test using Tester, and then click on the Tester’s Close button (X at top left corner of the Tester window), the entire instance of MT shuts down, not just the Tester, like it used to.

    So, for each test you run, you end up having to restart MT4, because the Tester button will force close the entire platform. This never used to happen before running Birt’s Patch and Script.

    Everything else seems to work great!

    Thanks.

    • #2082 written by birt April 5, 2012 (4 years ago)

      As I’m sure you’re aware, the Birt’s Patch script is not supported anymore.

      1. The configuration is saved regardless of whether your terminal is patched or not, it’s just the way MT4 behaves.

      2. The script has some limitations, the crash problem you noticed being among them. These issues are not present in the Tick Data Suite which completely replaces the script and brings some improvements.

      In other words, the Birt’s Patch script is free and will remain so, but it has some limitations, some problems and it is no longer supported. The Tick Data Suite is a paid product and it is fully supported, continuously updated and it doesn’t have the issues that the script has. Take a look at the feature matrix for more info.

  • #2083 written by iDouble April 6, 2012 (4 years ago)

    Hello Birt,

    I think I see, now. The Birt’s Patch is free and therefore, it crashes MT4 when you close Tester, whereas the TDS paid version does not crash MT4 upon closing the Tester.

    I think I understand, how that works now, after thinking about it for a moment. It actually makes perfectly good sense, when you stop to think about it. Crash MT4 with the free patch, and then solve that problem with TDS. Pure genius. I keep kicking myself for not coming up with the idea.

    Thank you very much, Birt. I appreciate the help and the time you take actually helping people, by not creating more problems for them to solve.

    I’ll be sure to check-out the Feature Matrix! I wouldn’t miss that for anything in the world at this point – especially, given the ease with which the Birt’s Patch has worked out for me. The entire process went very smoothly and everything worked exactly the way you said it would. MT4 crashes, but hey, its freeware – so I can’t complain!

    You turned out to be a real stand-up guy! Most forex retail websites, sell you a lot of headaches that don’t solve problems, but create problems. Then they turn around and try to sell you even more headaches, which cause even more problems. In the meantime, one forgets all about trading, given all the problems they encounter with what’s being offered online these days in the retail traders community.

    So, again – thank you for being stand-up!

    • #2084 written by birt April 6, 2012 (4 years ago)

      It’s all part of my plan to take over the world. It has nothing to do with the fact that there are limitations to what can be done from within an MQL script.

      Are you familiar with the proverb “never look a gift horse in the mouth”? If you wish to use the Birt’s patch script without any crash, you’re invited to use MT4 build 402, the script has no problems there. But wait, all the EAs compiled with build 405 or higher will not work in build 402 and will simply crash the terminal, with or without the script, so now you have a reason to also head to Metaquotes and tell them they’re stand-up. For the record, one of the reasons for creating the Tick Data Suite was that it was not possible to remove some crashes using MQL (e.g. the optimization crash or the crash when enabling the patch after a backtest).

      Anyway, nobody is forcing you to use the script if you have such a big problem with the crash upon closing the tester. Also, it’s open source with a GPL license – you are free to attempt fixing it if you wish or to find someone to fix it for you. After all, it’s just code, what could be so hard, right? Be sure to let me know if you succeed, I will gladly post it on the downloads page here.

  • #2085 written by iDouble April 6, 2012 (4 years ago)

    Birt,

    One last (before I go) quick question about back testing multiple time frames in general, if you don’t mind.

    I have created .HST files for every time frame that I need to test. I have placed those .HST files in the \history folder of MT4, as your docs suggest. So, my question is this, can I simply run an EA that is designed with Indicators and trade logic that uses all the time frames for which I have .HST files located in \history?

    I realize that the Tester is being fed .FXT files. How, does the Tester know to look at the \history folder, when one of the Indicators, or the trade logic used in the EA, needs to use data that comes from a different time frame?

    Thanks, again!
    iDouble

    • #2086 written by birt April 6, 2012 (4 years ago)

      The strategy tester uses the data in the HST files when you use an indicator so it doesn’t matter which timeframe the indicator is running on as long as you have the HST file for that timeframe. It’s only reading the ticks from the FXT file and it always supplies the current tick bid price as the close price for the current bar to allow indicators to also work properly with shift 0.

      That’s why it’s mandatory to have the HST files that are in sync with your FXT, otherwise the results would be all wrong.

  • #2087 written by iDouble April 6, 2012 (4 years ago)

    Of course, I meant:

    \Broker’s MT4 Name\history

    Is where all .HST files are located for all available time frames and for the entire range of data that was downloaded from TrueFx.

  • #2088 written by iDouble April 6, 2012 (4 years ago)

    Of course, I meant:

    \history\broker’s-name

    Sorry, for the confusion.

  • #2089 written by iDouble April 7, 2012 (4 years ago)

    Hey Birt,

    Thanks for the ADV on multi-time frame back testing. I was being intentionally sarcastic, but in a good way. I think you’ve done an outstanding job with the Patch and Script, and think you should be able to enjoy the benefits from any encapsulated obsolescence. And, whether such encapsulation was planned, or not planned, is perfectly ok and acceptable because the tool was free of charge.

    That’s why I said, I should have thought of it myself. If planned, it would be marketing genius. If not planned, it would be highly fortuitous at a minimum. Either way, GOOD JOB!

    I’m sticking with my plan to not allow Build 409 to auto-update and that has worked like a charm ever since. Besides, according to MetaQuotes, B409 was the last build for MT4 that would be updated. So, whatever build number is in play right now for MT4 – that’s it, according to their website/forum. They say that support for MT4’s last build would continue, however. But, my guess is that they will now turn all of their resources and attention, to MT5 and its significant array of problems.

    Oddly, none of the retail brokers that I consider worth heir weight in salt (and there are only two of them in retail Forex, as far as I am concerned) have told me they have any current plans for moving to MT5. So, I have no idea what MetaQuotes has been smoking, but if was the kind of guy who got high like that (I’m certainly not), I would absolutely want what they use – because that company’s decision to not make MT5 backwards compatible, set them up as being a MetaMeth Lab, as far as I am concerned – what a dumb decision. All that MT4 work that people have been doing for years, now down the tubes – if their broker decides to take the leap.

    I hope they all stay with MT4, just to teach MetaQuotes a good lesson.

    But, of course, I digress just a little.

    Anyway, rock solid work, Birt!

    • #2090 written by birt April 7, 2012 (4 years ago)

      I was of course being a bit sarcastic in my message, too. I even included an </sarcasm> but since the wordpress comments are not very smart, it thought it’s an HTML tag.

      Anyway, to make things clear, there were some very big changes from build 402 to build 405 which prompted me to completely rewrite the patch for that version (it’s quite visible in the script code). I did my best to make backtesting work, but this had a few rather nasty side effects (crashes) that could not be addressed by MQL code alone. Since backtesting works (which is the point of the script), I decided to release it as it was – most people don’t need optimizations and if one is careful, the crashes are rather easy to avoid. As it happens, there are many people who only need backtesting and who are content with build 409 (there’s really no incentive to get any newer build) so the script is still available and it works with the latest version of all the other free tools that you can find on this website (e.g. CSV2FXT). The Tick Data Suite uses a different approach and it is written from scratch with optimization in mind and designed to avoid any possible crashes such as those encountered with the script. Unfortunately, I lack the time to work on updating both so naturally the premium product is prioritized.

      Regarding MT5, it is kind of crap. From a programming point of view, its MQL is object oriented and kind of makes sense, but to be honest it’s a pain in the ass when compared to MT4 MQL which is a total breeze. From a broker’s point of view, it might make a lot of sense for US brokers, but I believe the major deterrent is the fact that there are not many EAs and indicators for it and upgrading from MT4 to MT5 costs 50k and I guess many brokers aren’t willing to just throw that money at Metaquotes for a piece of software that is at best decorative at the moment. My guess is that no MT4 broker is going to stop offering MT4 and fully migrate to MT5 anytime soon. This year has already seen 4 new MT4 builds (two of which had major bugs) so it looks like they’re still working on MT4 but they’re damn sloppy and it seems their release chain doesn’t include much of a testing phase.

  • #2091 written by Jacob April 7, 2012 (4 years ago)

    It seems that for the new built of MT4 419 your patch doesn’t work anymore … it is overwriting my tickfiles again 🙁
    Or am i just stupid?

    • #2092 written by birt April 7, 2012 (4 years ago)

      The Birt’s patch script only works up to build 409. The Tick Data Suite replaces it and works with newer builds.

  • #2093 written by Joao Santos April 8, 2012 (4 years ago)

    Oops! I now realize that I asked you to comment on a competitor’s product. Sorry about that slip of mine.

    • #2094 written by birt April 8, 2012 (4 years ago)

      Nah, Forex Tester is not exactly a competitor product as far as I’m concerned. They offer software that allows you to run backtests, but you can’t backtest MT4 EAs. They also offer tick data so, if anything, it’s a complementary product. Also, even if it was competition indeed, I wouldn’t have minded at all.

  • #2095 written by iDouble April 12, 2012 (4 years ago)

    Hey Birt,

    I agree. I don’t think that Metaquotes has even heard of QA/QC for the software development life cycle. In fact, they’ve probably never even heard of SDLC at all.

    I wish ProTrader had gotten their act together, instead of bailing-out on the small trader, in favor of developing broker products/services. I thought ProTrader really had a chance, and the mult-broker access from a single platform without programming, is the absolute perfect concept. But, ProTrader decided to go Hollywood, instead of sticking with the grass roots trader.

    The only hope right now, seems to be http://www.TradeInterceptor.com. A multi-broker/mutli-account access platform from a single desktop application. Built-in multi-time frame back testing and multi-time frame indicator overlay on the primary trade execution chart. I think it is java based (might be the only drawback) and they really do need to focus more on Westernizing their operations. They do have development road maps for supporting MT4/MQL, and I’ve been told that they will support existing MT4 EAs, when all is said and done. I don’t know their time tables, but NOW would not be soon enough.

    I’ve come across another question for you, if you don’t mind. Where do you go inside the MT4 install directory, to modify the Tester Chart Template that gets called to screen by the Back Tester?

    I always end up having to modify the standard back tester chart template, by pausing the back tester and selecting the template I need – going through the process of adding each indicator I want to visualize during the test. That’s a lot of wasted time. I’d like to be able to set one Tester Template, for each EA design – done.

    Thanks!
    iD

    • #2096 written by birt April 12, 2012 (4 years ago)

      The major disadvantage when it comes to such platforms is their lack of MQL support. It’s pretty hard to get a MT4 EA in ex4 form to work with a different platform, but if they manage that, their software will likely grow very popular.

      Regarding your template question, I’m afraid I don’t know, sorry.

  • #2097 written by Adam April 15, 2012 (4 years ago)

    Gday Birt,

    Ive some question relating the source of the spreads downloaded from the Jforex platform, especially as to how it relates to the variable spread in 99% backtesting.

    1) where would I find the find the feed that tick data is compiled from? I assume its simply via the Jforex platform – but Ive learnt not to assume anything – better to ask.

    Id like to compare the spreads – Jforex versus the broker I use – just an eyeball side by side. If the’re a big dispairty, then that throw out the results? (especially with tick scalper) ie if my broker had much lower (or higher) spreads?

    2) does the spread values used each tick 100% accurately reflect what they were at each tick ? or is there some forumula applied that creates the spread fluctuations ?

    I assume then if the EA had a spread filter and the spread rose above the max threshold, then the EA would not take the trade – correct?

    3) if the spread values are accurate tick by tick – why would anyone bother testing with a fixed spread? isnt the whole idea of tick data testing to most accurately reflect the real market?

    Thanks very much for your help, and please forgive me if these are questions already answered someplace on your website.

    Cheers, Adam 🙂

    • #2098 written by birt April 15, 2012 (4 years ago)

      I wrote a very long reply some 15 mins after you posted, forgot to hit “Post comment” and restarted the browser which is rather frustrating so I’m going to be shorter now.

      1. It’s from the Dukascopy servers so you’ll find it in JForex (demo is the same as live). If your broker’s spreads are very different, the backtest results might be misleading depending on the EA, of course. If that’s the case, you can try one of the other free tick data sources (Pepperstone, Oanda, etc.) – one of them might be closer to what you have.
      2. The spreads during a variable spread backtest reflect the exact difference between the ask and bid price from the tick data CSV used to generate the FXT. In other words, they’re 100% accurate. If your EA has a filter, trades will not be taken if the spread exceeds the threshold, assuming the spread is read by calculating the difference between the Ask and the Bid instead of using MarketInfo(Symbol(), MODE_SPREAD).
      3. Because spreads back in 2007 were much wider than what they are today. If the pair that your EA trades has a spread of e.g. max 4 nowadays, there’s not much point to backtest it during 2007 when the spread was as high as 10, especially if it’s a spread sensitive EA. Such real spread backtests usually represent a worst case scenario.
  • #2099 written by Murray April 15, 2012 (4 years ago)

    Hi Birt,

    I’d also like to know the answers to these questions Adam has asked. Especially the last one regarding why would anyone use fixed spreads when variable spreads is possible, especially for spike traders like MDP for instance.
    Regards,
    Murray

    • #2100 written by birt April 15, 2012 (4 years ago)

      In my opinion it’s close to useless to backtest spike traders such as MDP. Fixed spread or real spread, you still won’t get anywhere close to the results you’re getting on a live account.

      As for the answers for Adam, I suppose you read them above your post when the page refreshed 🙂

  • #2101 written by Adam April 19, 2012 (4 years ago)

    Gday Birt,

    Thanks very much for taking the time to provide a quality reply, really appreciated. The FX community is lucky to have you in the mix.

    And yes…don’t I know the frustration of losing a detailed reply sometimes ! grrr

    Anyways, Ive listed your 99% tick data suit/url page in NMi’s forum in case others want to go down the path of 99% testing.

    Thanks again, Birt.

    Cheers, Adam 🙂

  • #2102 written by Guest April 20, 2012 (4 years ago)

    Hi Birt, do you have any products for MT5? I am starting to code on the platform but I need a reliable data suite for it.

    • #2103 written by birt April 20, 2012 (4 years ago)

      Unfortunately, not at the moment and I’m currently not planning to release something like that. Perhaps in a couple of years if Metaquotes manages to shove it down the brokers’ throat and to make the EA developers start working with it. Right now, the platform is not popular at all and rather unfriendly to programmers. I haven’t seen a single EA released only in MT5 form or a single broker that as much as plans to drop MT4 support.

  • #2104 written by Renato April 25, 2012 (4 years ago)

    Hello Birt,
    just to inform you that the PROCESS component of the “dukascopy_php_scripts_v0.25” script produces the following error: failed to extract [EURUSD/2007/02/30/16h_ticks.bi5].

    I tried to locate the0.24 version which fixed a similar issue but I could not find it.

    Thanks and regards
    Renato

    • #2105 written by birt April 26, 2012 (4 years ago)

      I’m going to assume you’re using Windows since you didn’t specify otherwise.

      Did you copy the 7za executable in the same folder? Do you get a warning message when you run the script?

      If you have 7za.exe in there and you get no warning message when starting the script, it means that the bi5 file mentioned is broken. You have to delete it (just that file, not all the data) and run the download script again, it will fetch a new copy.

  • #2106 written by George April 25, 2012 (4 years ago)

    Hi Birt,

    I am using 1 hour and 1 minute period ema’s in my EA. I tried today and it seems to work fine with only 1 hour EMA. It seems the 1M and 1H ema’s at the same time won’t work.
    Multiple timeframes is allowed with the Birt’s pacth version ? I run CSVtoFXT program in all timeframes and all the fxt and hst files are stored in expert/files folder correctly.

    Your help will be highly appreciated. Many Thanks !!

    • #2107 written by birt April 26, 2012 (4 years ago)

      Yes, multiple timeframes are working correctly. You only need the FXT for the timeframe you are using in your backtest, the other FXT files are not necessary because the MT4 client only uses the HST files for multiple timeframe indicators.

      Anyway, please provide more details, what lead you to believe that it doesn’t seem to work?

  • #2108 written by george April 26, 2012 (4 years ago)

    Hi Birt,

    Thanks for your answer !

    My EA works with 2 ema’s : one on the current timeframe (30M,1H) and the other ema with 1 Minute period. I run the EA in a MT4 normal strategy tester under 30M timeframe and it works fine. This means that the signal based on the 30M and 1 minute ema opens the orders correctly.

    When I used the MT4 version with birt’s patch, it seems that the EA does not recognize the 1M ema indicator, since no orders are opened. If I delete the portion of the code corresponding to the 1M ema, the EA does open the orders. So we may conclude that the 1M period ema indicator is not recognized for any reason. If this is happening with other EA’s then this could cause misleading results.

    I am using Oanda v. 401, dukascopy Jforex data and Birt’s patch version for r. 401. All HST files for each period are in the history/Oanda practice folder.

    This is the portion of the code I have :

    ***** Current timefarme 30M EMA’s :
    SlowMAbar0 = iMA(NULL,0,SlowMAPeriod,SlowMAShift,SlowMAMethod,SlowMAAplliedPrice,MASignalBar);
    SlowMAbar1 = iMA(NULL,0,SlowMAPeriod,SlowMAShift,SlowMAMethod,SlowMAAplliedPrice,MASignalBar+1);
    SlowMAbar2 = iMA(NULL,0,SlowMAPeriod,SlowMAShift,SlowMAMethod,SlowMAAplliedPrice,MASignalBar+2);

    FastMAbar0 = iMA(NULL,0,FastMAPeriod,FastMAShift,FastMAMethod,FastMAAplliedPrice,MASignalBar);
    FastMAbar1 = iMA(NULL,0,FastMAPeriod,FastMAShift,FastMAMethod,FastMAAplliedPrice,MASignalBar+1);
    FastMAbar2 = iMA(NULL,0,FastMAPeriod,FastMAShift,FastMAMethod,FastMAAplliedPrice,MASignalBar+2);

    RSIbar0 = iRSI(NULL,0,RSIPeriod,RSIAppliedPrice,RSISignalBar);
    RSIbar1 = iRSI(NULL,0,RSIPeriod,RSIAppliedPrice,RSISignalBar+1);

    ****** 1 minute EMA:
    MSlowMAbar0 = iMA(NULL,1,SlowMAPeriod,SlowMAShift,0,SlowMAAplliedPrice,MASignalBar);
    MSlowMAbar1 = iMA(NULL,1,SlowMAPeriod,SlowMAShift,0,SlowMAAplliedPrice,MASignalBar+1);
    MFastMAbar0 = iMA(NULL,1,FastMAPeriod,FastMAShift,FastMAMethod,FastMAAplliedPrice,MASignalBar);
    MFastMAbar1 = iMA(NULL,1,FastMAPeriod,FastMAShift,FastMAMethod,FastMAAplliedPrice,MASignalBar+1);

    if ((FastMAbar1 SlowMAbar0) || (FastMAbar2 SlowMAbar0)) //&& RSIbar1 = RSISignalLevel)
    {
    if (TradeNotPlacedYet())
    {
    if (MFastMAbar1 MSlowMAbar0)
    {
    OpenBuyOrder(); ****** NOT OPENING ORDER HERE, IF I DELETE THE if (MFastM… portion it does open the order correctly…
    }
    }
    }
    if ((FastMAbar1 > SlowMAbar1 && FastMAbar0 SlowMAbar2 && FastMAbar0 RSISignalLevel && RSIbar0 MSlowMAbar1 && MFastMAbar0 < MSlowMAbar0)
    {
    OpenSellOrder();
    }
    }
    }

    • #2109 written by birt April 27, 2012 (4 years ago)

      First of all, please note that I am not offering any support for the Birt’s patch script.

      These being said, I suggest doing some logging to check what’s wrong in your particular case because I’m 100% certain that EAs using indicators on multiple timeframes work fine in tick data backtesting. In fact, I’ve checked it yesterday for a TDS customer and if you don’t want to take my word for it, write the following in an EA and backtest it on two different timeframes on ~1 month of data:


      int start()
      {
      if (iBars(Symbol(), PERIOD_M1) < 30) return (0); static datetime lastM5bar; if (lastM5bar != iTime(Symbol(), PERIOD_M5, 0)) { lastM5bar = iTime(Symbol(), PERIOD_M5, 0); double ima = iMA(Symbol(), PERIOD_M1, 30, 0, MODE_EMA, PRICE_WEIGHTED, 0); Print(DoubleToStr(ima, 5)); } return(0); }

      If you did everything right the results printed in the log should be identical. If they aren't, make sure that:

      • The same CSV was used for generating the FXT/HST for both timeframes.
      • The same GMT/DST settings were used.
      • The files were copied to the correct location.

      • #2110 written by George April 27, 2012 (4 years ago)

        Ok Birt,

        Surely I have something wrong with my installation or my code. I will check.
        Thanks for your time and your answer.
        George

        • #2111 written by George April 27, 2012 (4 years ago)

          Hi Birt,

          I repeated the CSV2FXT process for 1M and M30 timeframes and now is working fine. The code was ok.
          Before I did it with the option to create multiple FXT files in a single run.
          So something went wrong with that process… I guess.
          Just to give you a feedback.

          Thanks a lot.

      • #2112 written by George April 29, 2012 (4 years ago)

        Hi Birt,

        Well, I am getting the same problem again 🙁
        It’s weird….
        Yesterday was working fine after repeating the CSv2FXT process for M30 and M1 TF….
        Today I am getting 0 as the values for Ema’s indicators…so is not calculating values properly…
        The code is Ok.
        Any clue ? sorry to bother you about this…

        Thanks !!

        George

        • #2113 written by birt April 29, 2012 (4 years ago)

          Like I told you before, the Birt’s patch script is no longer supported. There must be a problem with your HST files but I couldn’t say what exactly.

  • #2114 written by Pirako April 27, 2012 (4 years ago)

    Hey birt 🙂
    ive been using your patch a lot and it worked just fine! (i dont mind the crashes when closing the tester.. anyways i thought might it be possible to have another patch which rlses the memory alteration to prevent from crashing?)

    But my actual question: Do you still have an installer for the old MT4 version compatible with the patch? I messed around with my installation and end up with a fresh install to 413 🙁 so the patch doesnt work anymore as stated out.

    Would appreciate a lil help 🙂

    @ geroge:

    i do multi TF backtesting with different indicators all the time.. no problems there 🙂 check ur code!

    greetings

    • #2115 written by Pirako April 28, 2012 (4 years ago)

      Found it myself- thanks! Since i got “serious” in EA programming i might switch to the walk forward analyzer soon! 🙂

  • #2116 written by Leo May 10, 2012 (4 years ago)

    Hi Birt,
    I have downloaded 1 month of data from Dukascopy website, then followed your instructions up till converting csv to fxt. All files are on the right folders.
    Could i then open my mt4 history center and import the 1 month data ? is it the fxt or csv file that i import ?
    Thank you so much for your great details in explaining everything….

    Cheers…

  • #2118 written by Adam May 16, 2012 (4 years ago)

    Gday Birt,

    I’ve downloaded and successfuly run 99% variable spread testing, using your script to choose true (use variable spread). When I uploaded data I inserted 0.6 commission, no problem.

    All works very well, thank you for the instruction.

    My question is if I set set the use variable spread script to : false, what spread does it use ?

    I assume setting it to false does fix the spread, and the tests shows a considerable better result, ie much easier to pass, NOT that I am looking for an easier test to pass. I am looking to compare a varaible spread test versus fixed spread.

    Thanks, your insights/advice is greatly valued.

    Cheers, Adam

    • #2119 written by birt May 16, 2012 (4 years ago)

      It is using the fixed spread saved in the file. Using fixed spread with an FXT built with real spread is a bad idea, you should build another FXT instead.

  • #2120 written by Murray May 28, 2012 (4 years ago)

    Hi Birt,

    Several brokers have now sent me an email along these lines.

    Starting on June 1, client MT4 terminals below build 416 will not be able to connect to trade servers, and thus updates and trading will be disabled. The only solution for traders with old terminal builds will be to download the latest version of MT4 and reinstall the terminal.

    This is a quick and easy process that we recommend you follow, regardless of your current MT4 build, to ensure you are running on the latest version of the IBFX MT4 platform with no interruption to your trading.

    So if I understand correctly this means that your Birts patch will no longer be able to be used on mt4 build 409 for testing. Are you going to be updating it to work on build 416 or later for those of us who only test occasionally? Or will the only solution be your paid product?

    Thanks,
    Murray

    • #2121 written by birt May 28, 2012 (4 years ago)

      If you want to keep using the free script, you can generate your FXT & HST files using the latest CSV2FXT and the latest MT4 build and then you can simply move them into a MT4 build 409 installation.

      I’m afraid there will not be any updates to that script, it is no longer supported.

      • #2122 written by Will June 3, 2012 (4 years ago)

        Problem with this birt is some EA’s require you to be connected to a broker, any way round this apart from data suite? Im not having a dig at you birt as you deserve paying something for your work. Im in no position to pay money just to try and optimise EA’s which I have carried out 100’s of backtests on before.

        • #2123 written by birt June 3, 2012 (4 years ago)

          No, sorry, I know of no workaround that would allow you to backtest EAs that require you to be connected to the broker.

  • #2124 written by FxMan May 28, 2012 (4 years ago)

    Hey birt,, you once talked of some ASM edits in your old page, to make new loader for variable spread backtesting, after 225 build (if one wants)

    As i need a 409 build multiloader with variable testing
    can you clarify whether i need to do those edits in multiloader for 409 (i.e generic loader) or on metatrader’s terminal.exe (of 409 build)

    I fired up w32dasm & even PE explorer disassm. but can’t find
    “00401323” to change
    “DA0D 36134000 FIQMUL DWORD PTR DS:[401336]” on that.

    Kindly help.
    Regards

    • #2125 written by birt May 28, 2012 (4 years ago)

      I can’t provide any support for that. In addition, the free script has variable spread support for build 409 and there is also the Tick Data Suite which works with all recent builds.

      • #2126 written by FxMan May 28, 2012 (4 years ago)

        OK,, i understand your issue and that support thing.
        But can’t understand the part “the free script has variable spread support for build 409”
        You mean the CSVtoFXT script? yes.. Old one also got the function. BUT one needs to have a multiloader with variable spread testing function to do multiple instances of testing with that FXT (i.e. when we do multiple backtests simultaneously )
        AFAI remember u built a 225 multiloader named “metatrader.4.build.255-multiloader+spread” for 225 build. There after nothing came and you also said
        “I don’t think I’ll be making one for 226 anytime soon”

        So that means for future builds too it never came out. Or i am doing/using something wrong?

        Regards

        • #2127 written by birt May 28, 2012 (4 years ago)

          There have been a lot of changes since then. Please take the time to re-read the guides because pretty much everything has changed; you no longer need a loader at all (unless you want to run multiple instances from the same folder).

  • #2128 written by Riaan June 8, 2012 (4 years ago)

    Please make sure you do not buy a subscription from FXTickData for your your tick data. They “claim” that you can do tick data back testing with free tools and then point you to Bert’s website. I got “caught” by this and I don’t want anybody else go get caught by them! Rather just buy Bert Tick suite as it IS the best and only tool to do tick data tests with!

    • #2129 written by birt June 8, 2012 (4 years ago)

      I had no idea about that, but I find it rather funny. Anyway, I guess they’re right to a certain extent – it’s still possible to use the free script with MT4 build 409, but it’s rather complicated to get it done.

  • #2130 written by rajiv June 10, 2012 (4 years ago)

    birt 99% is best
    what if we test EA on 90% molding quality ,is it simulated like as 25%

    • #2131 written by birt June 10, 2012 (4 years ago)

      I’m not sure what your question is (if any) but you get 25% modeling quality when backtesting an EA on an M1 timeframe using history center data.

  • #2132 written by JBente July 10, 2012 (4 years ago)

    Hello, I read the posts and wanted to ask you something:
    – Do you do the optimization with check points and the backtesting with each tick?
    – How many operations do you take to optimize?
    – Are the Dukascopy´s data based in GMT 0 with no DST?
    Thank you very much.

    • #2133 written by birt July 10, 2012 (4 years ago)

      1. I don’t really do much optimization myself. If I were, I’d do it using control points where possible (non-scalping strategies).

      2. Since I don’t do many optimizations, I don’t have a norm.

      3. Yes, the Dukascopy tick data is using GMT 0 and no DST.

  • #2134 written by JBente July 10, 2012 (4 years ago)

    Thanks, and with scalping strategies ¿does you make the optimization with every tick?
    Can you do optimization with each tick using the birt´s patch?

  • #2135 written by birt July 11, 2012 (4 years ago)

    Like I said, I don’t really do optimizations often but if I had to do it for scalping strategies, I would likely use tick data at least in some of the phases if not all.

    Tick data optimizations are not working with the birt’s patch script. However, they are working fine with the Tick Data Suite.

  • #2136 written by Sam Adams July 11, 2012 (4 years ago)

    Hello Birt,
    I’ve had tick data suite for some time now and all of a sudden when i set up a platform it would display the blue shield, now it displays a file sheet. Have i done something wrong or is this what to expect now? I am running several platforms for optimizing and back testing, just an fyi.

    • #2137 written by birt July 11, 2012 (4 years ago)

      Weird. If it works fine (meaning you can start it via the shortcut), there might be an issue with your system’s icon setup.

  • #2138 written by Luca July 13, 2012 (4 years ago)

    Hi Birt !
    Is it possible to use the tick data I collect from my broker to be converted in a suitable format to be used for backtesting ?
    If yes, in what format and with what fields I need to collect the data ?
    Thanks !

    Luca.

    • #2139 written by birt July 18, 2012 (4 years ago)

      Yes, it’s possible. You need something like:
      DD.MM.YYYY HH:MM:SS,Bid,Ask

      You can change the date format but make sure the date and time are in the same field. You can also change the order of Bid and Ask.

  • #2140 written by sean July 17, 2012 (4 years ago)

    Hi Birt, when I run birt’s patch I get an error dialog “can’t find _fseeki64()”
    I am running build 225 on windows xp (i.e. x86) but have a new version of the run time dll installed.

    • #2141 written by birt July 17, 2012 (4 years ago)

      Please note that support is no longer provided for that. Also, as specified in several places, the 2GB limitation removal will not work on XP with the free script.

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <s> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top