In general, backtesting using the data from the MT4 history center might be ok for EAs that are not scalping or pip hunting. If you’re dealing with such an EA or any kind of EA that closes trades within 1-15 pips, though, even the smallest feed differences might have a very large impact.

The issue is caused by the Metatrader terminal not having access to the real tick data, but only to minute bar data in the best case, which forces it to give your strategy backtest “false” ticks, generated through a process of interpolation of the data for the smallest timeframe available. This is most likely not important to an EA that uses a stoploss and takeprofit target of over 100 pips, but in the case of EAs that attempt to scalp a few pips here and there, your backtest could be totally misleading.

So, it is very important to try testing using data with a quality as high as possible, which is why I put together some resources, all of which I use in my backtesting when needed.

Dukascopy tick data

The broker

Dukascopy is universally acclaimed as one of the best brokers nowadays. Personally, I’ve never had anything to do with them so I cannot confirm or deny that, but many people classify it as “the best ECN broker” and I have yet to read bad things about them. From my point of view, their major drawbacks are the lack of a MT4 client (it’s possible to run MT4 EAs, but very tricky) and a very high minimum deposit limit.

The data

Unlike many other brokers or data feed providers, Dukascopy offers free tick data spanning from April 2007 (for most currencies) to now. Despite being offered at no cost, it’s commercial-quality data and it actually approaches the 99% quality displayed in the strategy tests (which, by the way, is just a number in the FXT header). From what I can tell, it is updated hourly and the source is none other than the Dukascopy data feed server. Bottom line is: I couldn’t wish for a better data quality.

The tools

Since the data is downloaded in files that span only 1 hour, some tools are necessary to download and parse it. I’m a fan of the PHP simplicity, so I chose that to write the scripts. They’re not commercial quality code, but they work. Feel free to do whatever you want with the code, but don’t forget to give credit where it’s due.

Special note: these scripts are provided for the tech-savvy people who want to keep really up-to-date on their tick data. If you’ve no idea where to start with them, just ignore them and skip to the Prepared data section below.

You can download the archive here.

Edit 16.11.2009: updated the archive with the fix Futron posted in the comments section.

Edit 30.12.2009: updated the archive to fix a minor bug.

Edit 21.04.2010: updated the process script to resume if the output file exists, effectively allowing incremental CSV generation.

You will find 3 scripts inside:

  • A script for downloading the Dukascopy data, suggestively named “download_dukascopy_data.php”. As a courtesy for Dukascopy who is graciously providing free data, even though the script doesn’t download existing data, it still requests missing files, so, to avoid stressing their server, please set the dates in the $currencies array at the beginning of the script to the date of the last download; they’re using the standard linux seconds-since-1970 format, look up mktime to figure it out.
  • A script for processing the downloaded data, which assumes that it’s in the same directory with the previous script and that the data was downloaded there (process_dukascopy_data.php); this one needs some parameters, run it without any for a description or check out the next script
  • A small linux shell script that will process all the downloaded data

A minimum requirement to run these scripts is a command line PHP with the CURL extension. If you’re using a windows box, I’m pretty sure there are installer packages with it, use the mighty Google to find one. On recent OS X releases it’s installed by default while on Linux distributions it might or might not be installed, but if it’s not, it’s typically easy to do it. I’ve only tested on Linux, but I don’t see why it shouldn’t work on other setups.

Prepared data

Due to the size of these files, it’s not feasible for me to host them, so I decided to upload them as torrents. To download these files, you will need a torrent client. For windows, I can recommend µTorrent – it’s very good, small and reliable torrent manager.
Please try to seed everything you download at least to 1.0 ratio.
The tracker hosting the torrent files is Legit Torrents, a tracker for legal torrents just like its name suggests. I thank them by these means for providing tracking services for the tick data torrents.
As a last piece of info before the links, I will update the files (by seeding new torrents) as my time permits, but, honestly, managing the torrents for each pair is a pain in the ass, which is why you shouldn’t expect them to be updated more than once per month.
To download the torrent files, just follow the links and click the torrent link in there.
It has been brought to my attention by a friend that Dukascopy prohibits redistributing the data, so I removed the torrent files, even though I think they are just provisioning against commercial redistribution. I will add a Windows do-it-yourself section instead.

Windows download & convert to CSV how-to

Start by going to the windows PHP download section and fetching the latest binary version as a zip file.
Once you’ve done that, unpack it to c:\php\ and also unpack the scripts from dukascopy_php_scripts.zip in the same directory.
Rename c:\php\php.ini-development to c:\php\php.ini.
Edit c:\php\php.ini, search for
;extension=php_curl.dll
and remove the semicolon in front of the line and add an “ext/” in front of “php_curl.dll” so that it looks like this:
extension=ext/php_curl.dll
Save the file and exit.
Edit 21.04.2010: if you run into a zip error and your PHP installation has an ext/php_zip.dll, apply the above for extension=ext/php_zip.dll.
Click start->run and type
cmd
then click ok (or alternatively type
cmd
then hit enter in the windows 7/vista “search programs and files” box in the start menu).
Type
cd \php
in the command window.
Type
php download_dukascopy_data.php
Have a coffee. Have another coffee. Go sleep. Go to work. Go to the gym. Go to a club. Wait some more. I’m not kidding, it takes ages. You can check the progress by watching the currency pair directories get filled. You can change the array at the beginning of the file to switch the order or to remove some pairs you don’t want. The number next to each pair is the timestamp at which to start downloading. If you get any strange errors, run the process again when it’s finished – it will only download any files that were missed in the first step due to network errors.
Edit 23.11.2009: if you want to easily convert from a regular date to a unix timestamp, you can use Epoch Converter, a very easy to use online tool.
When the download is finished, assuming you wanted to get the EURUSD data up to 01.11.2009, you’d type
php process_dukascopy_data.php EURUSD 200702 200911 EURUSD.csv
and the output will be placed in EURUSD.csv.
Alternatively, you can type
process.bat
which will batch process all the currency data. It’s mostly safe to ignore the error spam at this step.
This should be it, if everything went ok you should have your CSV files in the same c:\php folder.
Warning: make sure you have enough space on your harddisk. The downloaded tick files have over 7.5 GB while the CSV files have close to 60 GB at the time of this writing.

Using the data

Now, in order to use the CSV files, you need to perform some steps

  1. Convert the data to FXT & HST
  2. Copy the FXT & HST to the correct folders
  3. Run MT4 with a loader to allow loading your custom FXT file, then start backtesting

I’ll go through each step below:

Converting the data to an MT4 format

To convert the data to an FXT file (and the accompanying HST files) usable in Metatrader 4, you need a new script that you can download here.

This script is based on GainData2fxt.mq4, an older script by an author that I couldn’t determine and on FXTHeader.mqh, written by stringo@codebase.mql4.com. The first one has suffered major modifications at my hand, while the second one was only modified here and there to adapt it to the updated FXT header structure.

Edit 21.04.2010: this script has been updated to allow switching the GMT offset of the resulting FXT & HST files and to allow exporting the real spread in the file.

You have to perform the following steps in order to convert the CSV data:

  • Copy the files from the dukascopy_mql_scripts_22.04.2010.zip archive to your MT4 installation. There’s a directory structure inside, make sure they land in the proper places
  • Move the tick data file (the CSV file) to experts\files
  • Open a chart for the pair that you have data for (if you have EURUSD.csv you MUST open an EURUSD chart)
  • Select the timeframe that you wish to generate the FXT for (if you want to backtest on H1, then select H1 as the chart timeframe)
  • Double click the Dukascopy2FXT script in the navigation panel (it’s in the scripts section) and configure the parameters in the window that pops up.
  • Configure the parameters.
    For CsvFile you don’t need to write anything if the file is named just like the symbol and has a CSV extension; otherwise, just type the file name there.
    Select “true” for CreateHst only if you want the script to create HST files. You do not need to perform this step more than once (leave it on “false” if you’re regenerating data for a different timeframe), because if this parameter is set to true, history will be generated for the whole span of the CSV file and for all timeframes, regardless of the start/end date you configure (note that the FXT file is still limited by those).
    By using the Spread parameter, you can configure the desired spread of the FXT file. If left at 0, it will use the current spread that your broker has. A special note here would be that many brokers are widening their spreads during the weekends, so pay close attention. If set to anything other than zero, it will use the number as the spread; if your broker is a 5 digit broker, you have to enter 20 for a spread of 2 pips, while if your broker is a 4 digit broker, you would just have to enter 2. Small trick to change the spread without regenerating an FXT: open the existing FXT file with a hex editor; the spread is at offset 0xFC.
    The StartDate and EndDate fields control the span of the FXT file. There is a programming design bug in MT4, where the client is unable to read FXT files larger than 2GB and seeing that the tick data is rather huge, it’s very likely that your resulting file might be larger than that if you don’t pay attention. I added a “block” so the creation of the FXT file stops at 2GB, but keep in mind that if you don’t use set limits, you will probably not get the resulting data range that you desire. A good rule of the thumb is to use ~12 month periods for EURUSD and 16 month periods for any other currency. If the currency CSV file is below 2GB, most likely your resulting FXT file will be below 2GB as well and you’re probably safe to generate it for the whole range available in the CSV. The format of the StartDate and EndDate fields is YYYY.MM.DD and if left blank the script will just use the earliest, respectively the latest date available in the CSV file.
  • Click “Ok”. Once you do that, the data generation process will start and it will typically take 5 to 30 minutes, depending on the data range and volume, possibly even more if you’re using a slow machine. When the processing is finished, you will get an alert and some data will be printed in the Experts log.
  • Some people experienced miscellaneous issues. If your particular case is not listed here, feel free to drop a comment, I might be able to help.
    Among the common problems that the people encounter it’s worth mentioning the Windows 7 / Windows Vista users that run MT4 from their Program Files folder and that are unable to find the resulting files. The issue is caused by the fact that they have UAC enabled and these operating systems use folder virtualization. For Windows 7, the resulting files can be found in c:\ProgramData\ typically, while for Windows Vista they can be found in c:\Users\username\AppData\Local\VirtualStore\Program Files\. If they’re not there, just search for *.FXT, they must be somewhere. To get around this problem, either copy your MT4 folder in a location that’s not protected by UAC or simply disable UAC (type UAC in the start->search box thing and follow the on-screen process).
    Another issue experienced by some users is the lack of a MIN_LOT and LOT_STEP for the symbol in the resulted FXT file. This is caused by generating the FXT file with a MT4 client that has never been connected to a broker. When generating a file, do that on an MT4 client that has been connected to your broker at least once. Aka don’t use a “virgin” installation that has never seen a demo account.

Copying the data to the correct location

After the script above has finished running, you will end up with a lot of files in your “experts\files” folder. At this point, exit the MT4 terminal then proceed to moving any .HST files from “experts\files” to “history\your_server_name”. Pay very close attention if you have multiple server directories in your “history” folder – you will have to move them into the one that’s correct for active account!
Note: unless you modify the CSV or get a new one, the HST files only need to be generated and moved once.
After moving the HST files, move the generated FXT file from “experts\files” to “tester\history”.

Using the data in a backtest

Since the recent MT4 clients no longer offer the ability to use custom FXT files, a special loader is needed. You can download it here, it works only with the MT4 version commonly known as 225 (version 4.00 build 225).
To use the files you created in a backtest, you will have to copy the exe inside fxt_loader.zip into the folder of your MT4 client and instead of running terminal.exe, run “fxt loader.exe”. If you’re worried about running executable files and you can do a manual patch, here are the offsets and data:

hex offset -> old byte -> new byte
1402FC -> EE -> 00
1402FD -> 02 -> 00
14041B -> 9D -> 00
14041C -> 01 -> 00
140521 -> 97 -> 00

What the loader does is disabling the MT4 default behavior to regenerate the backtest files every time you start a new test – so, if a file is already there, a new one will not be generated. By placing the FXT file you generated with the script in the tester\history folder, you ensure that it’s the file it is going to use for your backtest; of course, the selected backtest currency pair and timeframe must coincide with what you generated the FXT with.

Edit 24.09.2009: As an alternative to the loader above, you can use this loader which, in addition to what the other one does, will also let you run multiple instances of the terminal from the same installation folder. Read the comments below for more info and the patched offsets.

Edit 29.03.2010: Added a build 226 loader. It also includes an MQL script for the people who don’t want to run EXE files or who have particularly dumb antivirus packages; if you want to use the script version, start MT4 then put the script on any chart to enable the FXT functionality. However, keep in mind that while you can start more than an instance of the same MT4 client with the loader, you can’t do so with the script. If you want the patch offsets, they can be found in the script. Note that running the script on a build other than 226 (225 for example) will surely make it crash, so double check what build you’re running before you use it. On a separate note, you can make a similar script for build 225 by using the offsets posted above.

How to verify it worked?

That’s easy: if your test is running and the file in tester\history was not recreated, it usually means everything is ok.

GMT offset

This is a sensitive issue for many EAs. The Dukascopy data seems to be GMT-based all around the year, so the GMT offset that you need to use is 0. As of 21.04.2010, the Dukascopy and Oanda MQL scripts have an external parameter that lets you specify the GMT offset you would like for the resulting data, both HST and FXT.

Gain Capital tick data

Gain Capital is the company behind forex.com. From what I hear, they’re a decent broker, but not one I would recommend. They’re the kind of “pushy” broker, I remember opening a demo account a few years ago and then getting several emails along the lines of “hi, are you going to open a live account already?” during the course of the next few months. What’s actually scary is that the emails were not automated.
Anyway, they also offer some free tick data. If you’re feeling a bit masochistic and want to play with it, I recommend using a download manager to fetch it. Fair warning: the quality is pretty bad – not only is the data missing large chunks of time, but it’s also sporting misaligned periods (e.g. you can sometimes find data from the previous week in the file for the current week).
Still, before finding the awesome Dukascopy data, I used this for testing so I perfected some scripts for it, available here.
Inside you will find:

  • A script to process the data into a CSV. This script is heavily based on the original by carloss from the indo-investasi forum.
  • A script to somewhat filter & sort the resulted CSV.

Since these are totally obsolete to me, I’m not going to provide more details and you’re going to have to figure out how they work. Shouldn’t be too hard, though.

Oanda tick data

If you’re one of the lucky people who can get the Oanda tick data (it’s only available to people who have an account there funded with over 1k USD), get it in CSV format and use the Oanda2FXT MQL script. It works just like the Dukascopy2FXT script described above.

Variable spread backtesting

If you want your backtest to be even more accurate, as of 21.04.2010 you also have the option to use the real spread from the tick data. All you have to do is:

1. Make sure you’re using the latest Dukascopy MQL script or the latest Oanda MQL script.

2. When running the script, set UseRealSpread to true.

3. When loading the terminal, use the variable spread loader (which is different from the other loaders).

Note that as of now, this only works on Metatrader 4 build 225. I don’t think I’ll be making one for 226 anytime soon, so if there’s anyone who fancies asm and thinks they can handle porting it to 226, here’s what I changed:

00401323 DA0D 36134000 FIMUL DWORD PTR DS:[401336]
00401329 DB17 FIST DWORD PTR DS:[EDI]
0040132B E9 26020000 JMP terminal.00401556

00401336 A0860100 //100000
0040133A ACC52737 //0.00001

00401556 DA27 FISUB DWORD PTR DS:[EDI]
00401558 D815 3A134000 FCOM DWORD PTR DS:[40133A]
0040155E EB 12 JMP SHORT terminal.00401572

00401572 DFE0 FSTSW AX
00401574 F6C4 01 TEST AH,1
00401577 E9 E5020000 JMP terminal.00401861

00401861 DC42 1C FADD QWORD PTR DS:[EDX+1C]
00401864 74 06 JE SHORT terminal.0040186C
00401866 DC83 20030000 FADD QWORD PTR DS:[EBX+320]
0040186C C3 RETN

00541209 E8 1501ECFF CALL terminal.00401323
0054120E 90 NOP

Ninjatrader export

A kind soul nicknamed xTrader1 has modified the process php script and made it export data that can be used in Ninjatrader. Quoting from his comment that is also available below if you’re into digging:

I tweaked your PHP script “process_dukascopy_data.php” in such a way, that it creates an original CSV file, but in addition 2 files Bid and Ask prices in ready to import NinjaTrader format.

He’s also been decent enough to send me the script and give me permission to host it here. Just as a disclaimer, I haven’t actually tested it so if it kills your dog or eats your babies I can’t be held responsible. Without further ado, I give link you the ninjatrader parse script. For some reason that I can’t explain, I got quite a kick out of naming the archive “dukascopy ninja script”.

Download links

For easier reference, here you go:
Dukascopy data download / parse scripts
Dukascopy MQL process scripts
MT4 build 225 FXT loader
MT4 build 225 FXT multiloader
MT4 build 225 FXT multiloader with variable spread
MT4 build 226 FXT multiloader and script
Gain Capital data tools
Oanda MQL Process script

VN:F [1.8.1_1037]
Rating: 9.9/10 (26 votes cast)
VN:F [1.8.1_1037]
Rating: +30 (from 30 votes)
Tick data9.91026