The article you are about to read is a detailed description of how to acquire free historical Forex tick data and use it in backtesting MT4 EAs to obtain a 99% modeling quality. If you don’t understand even one of the terms in the previous sentence, it’s probably best to either stop reading now or just purchase the Metatrader Backtesting and Optimization Course.

In general, backtesting using the data from the MT4 history center might be ok for EAs that are not scalping or pip hunting. If you’re dealing with such an EA or any kind of EA that closes trades within 1-15 pips, though, even the smallest feed differences might have a very large impact.

The issue is caused by the Metatrader terminal not having access to the real tick data, but only to minute bar data in the best case, which forces it to give your strategy backtest “false” ticks, generated through a process of interpolation of the data for the smallest timeframe available. This is most likely not important to an EA that uses a stoploss and takeprofit target of over 100 pips, but in the case of EAs that attempt to scalp a few pips here and there, your backtest could be totally misleading.

So, it is very important to try testing using data with a quality as high as possible, which is why I put together some resources, all of which I use in my backtesting when needed.

The article you are reading requires a fair bit of computer proficiency. If you need more detailed instructions or if you run into issues not dealt with in the comments section, I strongly suggest heading to the Guide to Getting Rich with Forex Robots. The guys there have put together some very nice video material detailing 99% backtesting step by step, designed to help the users who are not very experienced when it comes to computers, together with a FAQ with the most common problems you may encounter and with screenshots for each step in the video.

I’ll be honest with you: at first, when I saw the name of the product I thought it would be some bullshit, but then I had to check it out (due to the 99% tools used, part of which are originating from me) and I found it so useful for the less seasoned people that I ended up becoming an affiliate. In addition to the 99% backtesting info, they put together a lot of useful info about EAs, starting from the basics of backtesting, going in detail into money management strategies and also sporting a very thick “how to spot scam EAs” section that will give you a solid idea of what’s happening with many commercial EAs and what are the processes behind them. In a nutshell, if you’re rather new to Forex or if you want a peek behind the curtains of selling EAs, this guide will probably save you a ton of money. On top of it all, they have a very good support system (I’ve had to use it myself) and they are helping people that are stuck with 99% modeling quality backtesting or with other aspects of the guide. The only major flaw I can find is its name: I would’ve rather called it “Guide to avoid Losing Tons of Money with Forex Robots”; other than that, it’s well worth the $87.

These being said, let’s delve into the procedures.

Dukascopy tick data

The broker

Dukascopy is universally acclaimed as one of the best brokers nowadays. Personally, I’ve never had a live account with them so I cannot confirm or deny that, but many people classify it as “the best ECN broker” and I have yet to read bad things about them. From my point of view, their major drawbacks are the lack of a MT4 client (it’s possible to run MT4 EAs, but very tricky) and a rather high minimum deposit limit ($5000 last I checked). However, if you’re looking for a really solid ECN broker and EAs are not your kind of cookie, look no further.

The data

Unlike many other brokers or data feed providers, Dukascopy offers free tick data spanning from April 2007 (for most currencies) to now. Despite being offered at no cost, it’s commercial-quality data and it actually approaches the 99% quality displayed in the strategy tests (which, by the way, is just a number in the FXT header). From what I can tell, it is updated hourly and the source is none other than the Dukascopy data feed server. Bottom line is: I couldn’t wish for a better data quality.

The easy way

If you’d rather skip the whole PHP process described in detail below, there’s an easier way, but it requires you to register a demo account. I’ve used it in the past, but for some reason it didn’t occur to me that I should also post it here for the readers’ convenience until one of the blog readers pointed it out.

So, begin by registering a demo account with Dukascopy and start the JForex platform (or register a live account, the data process is the same). Login using the data in the email you received, then go to the Tools menu and click Historical Tester. In the lower part of the window, the Historical Tester interface should appear; from now on, everything you have to do happens in that part of the window. First, select the symbol that you want the tick data for in the combo box on the left side (EUR/USD is selected by default). If you need the data for any additional symbols, click the small plus sign next to the combo box, which will pop up an Instrument Selector window. Select any other additional symbols that you want the tick data for and click Ok. Ignore the combo box to the right of the plus sign and click the calendar button to its right. Select 2007.03.01 as the start date (no tick data is available before that date) then select an end date of your choice. Note that if you select today’s date you might run into download errors as not all the data is available yet. After you’re done with the period selection, make sure “Ticks” is selected in the combo box below the symbol and click the “Get Data” button.

Very important: if you are not using a locale that has a dot (“.”) as decimal separator, the FXT script will not work with your CSV. If you get broken FXT files, it’s most likely because your decimal separator is a comma and to solve this problem you have to switch to an English locale (via Control Panel -> Regional Settings or Control Panel -> Region and Language), restart JForex, export the CSV files, then switch back to your original locale. If you don’t want to switch your locale, the alternative solution is to use the PHP method below.

At this point, you’re going to have to patiently wait until the progress indicator slowly (exactly how slowly depends on the amount of data you selected) crawls to 100%. If it says “downloading failed”, try again; if it still fails, try a shorter period of time or symbol by symbol in case you selected multiple symbols. If it still fails and you can’t figure it out, just use the PHP method described below.

Assuming the download went fine, you can click the Save Data button and choose the path and the file names for the CSV files for each symbol. Now you can use the JForex2FXT script included in the scripts archive script to convert the data into an FXT; it works just like the Dukascopy2FXT script and you can find instructions on its usage in the “Converting the data to an MT4 format” paragraph below – all you need to do is use JForex2FXT instead of Dukascopy2FXT).

Note: do not confuse JForex2FXT with Dukascopy2FXT, they are different, even though the input parameters are looking identical.

The alternative

Warning: if using this method results in your backtests abruptly stopping in the middle, please try a different method (PHP or JForex).

Tim Cragg started writing an ambitious .NET application named DukasCopier that aims to handle the whole process in an easy, user-friendly fashion. The application is a work in progress and it currently handles downloading the data and exporting the CSV at the same time, thus saving a lot of time when compared to my PHP scripts. Ultimately, it aims to also export the HST and FXT files but those features are not yet available. For those not inclined to sign up for a JForex account and not eager to get into the whole PHP script thing, this is the easiest solution.

The application is compatible with my PHP scripts so you will have to use the Dukascopy2FXT script to further process the resulted CSV files. It is also able to use any existing data downloaded by the PHP scripts by using the same directory structure, saving you the pain of a new download.

Tim asked me to host the application until he finds a more permanent home for it. I’ve tested it a bit and it seems to work just fine, so you can get it at the link above. I will try to update it if and when Tim provides updates. I’m not listing Tim’s email here due to spam bots; if you want to contact him, his email address is listed in the about page of the DukasCopier.

Please note that even though I tested the application a bit, I take no responsibility whatsoever for it. Should it decide to engage in any kind of intercourse with your computer or otherwise misbehave, feel free to post a comment but keep in mind that DukasCopier is offered on an as-is basis, with no express or implied warranty of any kind.

You will need the .NET Framework 4.0 to run the application.

The complicated way

Over the web, the Dukascopy data is available for download in files that span only 1 hour, so it follows that some tools are necessary to download and parse it if we don’t want to install JForex. I’m a fan of the PHP simplicity, so I chose that to write the scripts. They’re not commercial quality code, but they work. Feel free to do whatever you want with the code, but don’t forget to give credit where it’s due.

You can download the archive here.

You will find 4 scripts inside:

  • A script for downloading the Dukascopy data, suggestively named “download_dukascopy_data.php”. As a courtesy for Dukascopy who is graciously providing free data, even though the script doesn’t download existing data, it still requests missing files, so, to avoid stressing their server, please set the dates in the $currencies array at the beginning of the script to the date of the last download; they’re using the standard linux seconds-since-1970 format, look up mktime to figure it out. If you want to easily convert from a regular date to such a unix timestamp, you can use Epoch Converter, a very easy to use online tool.
  • A script for processing the downloaded data, which assumes that it’s in the same directory with the previous script and that the data was downloaded there (process_dukascopy_data.php); this one needs some parameters, run it without any for a description or check out the next script.
  • A small shell script that will process all the downloaded data available in .bat form for windows and .sh form for linux.
  • As of 15.01.2011, you can also find a script that downloads candle data.

Windows download & convert to CSV how-to

Start by going to the windows PHP download section and fetching the latest binary version as a zip file.
Once you’ve done that, unpack it to c:\php\ and also unpack the scripts from dukascopy_php_scripts.zip in the same directory.
Rename c:\php\php.ini-development to c:\php\php.ini.
Edit c:\php\php.ini, search for
;extension=php_curl.dll
and remove the semicolon in front of the line and add an “ext/” in front of “php_curl.dll” so that it looks like this:
extension=ext/php_curl.dll
Save the file and exit.
If you run into a zip error and your PHP installation has an ext/php_zip.dll, also apply the method above for extension=ext/php_zip.dll.
Click start->run and type
cmd
then click ok (or alternatively type
cmd
then hit enter in the windows 7/vista “search programs and files” box in the start menu).
Type
cd \php
in the command window.
Type
php download_dukascopy_data.php
Have a coffee. Have another coffee. Go sleep. Go to work. Go to the gym. Go to a club. Wait some more. I’m not kidding, it takes ages. You can check the progress by watching the currency pair directories get filled. You can change the array at the beginning of the file to switch the order or to remove some pairs you don’t want. The number next to each pair is the timestamp at which to start downloading. If you get any strange errors, run the process again when it’s finished – it will only download any files that were missed in the first step due to network errors.

When the download is finished, assuming you wanted to get the EURUSD data up to 01.11.2009, you’d type
php process_dukascopy_data.php EURUSD 200702 200911 EURUSD.csv
and the output will be placed in EURUSD.csv.
Alternatively, you can type
process.bat
which will batch process all the currency data. It’s mostly safe to ignore the error spam at this step. Note: if you use process.bat or process.sh, you might have to update the ending dates in them to get the full data range!

This should be it, if everything went ok you should have your CSV files in the same c:\php folder.

Edit 07.02.2011: it appears that a bug in recent versions of PHP results in a CLI crash. If your processing suddenly stops, you are most likely suffering from this issue. Since it will likely be a while until the PHP team publishes a release with the bug fixed, I made an alternate version of the process PHP script that implements a workaround for this particular problem.

Warning: make sure you have enough space on your harddisk. The downloaded tick files have over 7.5 GB while the CSV files have close to 60 GB at the time of this writing.

Using the data

Edit 09.01.2011: I’ve made some major changes to the loaders and export scripts. For those who are familiar with the old ‘toolchain’, the following aspects have changed:

  • The loader is only required if you want to start more than an instance of MT4 from the same folder.
  • The patching is now done via “birt’s patch.mq4″ which goes into your scripts folder. Run it on a chart and choose what you want it to patch.
  • It is now possible to use files over 2GB (up to 4GB) but only on Windows 7, Vista and Server 2008 (and above). This will not work on Windows XP or Server 2003 or any other earlier version of the OS.
  • Variable spread can be used in any MT4 build (225-229).
  • The old variable spread FXT format is now invalid. The new variable spread FXT format uses the volume (thanks Hyperdimension for the suggestion). The drawback is that until you restart the MT4 terminal, you can’t backtest regular FXT files after the variable spread patch is applied. Note: all the patches are applied by the patch script at runtime, in memory, so they’re not permanent.
  • You can pad the spread when using variable spread.
  • You can configure a commission in pips.
  • In the FXT script parameters, all the spread and commission values are in pips. This means that no matter if your broker is 4 or 5 digits, if you want a spread of 2 pips, you write 2.
  • The generated FXT files are automatically set read-only.

The rest of the article is going to be updated to reflect the current state of things.

Now, in order to use the CSV files, you need to perform some steps

  1. Convert the data to FXT & HST
  2. Copy the FXT & HST to the correct folders
  3. Use a script-patch in MT4 to allow loading your custom FXT file, then start backtesting

I’ll go through each step below:

Converting the data to an MT4 format

To convert the data to an FXT file (and the accompanying HST files) usable in Metatrader 4, you need a few scripts that you can download here.
Depending on what source data you have at this point, you need to use the scripts as follows:

  • If you created the CSV using JForex, you need to use JForex2FXT.
  • If you used the PHP scripts to create the CSV, you have to use Dukascopy2FXT.
  • If you have an Oanda CSV file, you have to use either Oanda2FXT or Oanda2FXT-alternate, whichever works for you. The reason there are two scripts for Oanda is that I’ve seen tick files from them with two different delimiters (space and comma).

Edit 15.01.2011: It has come to my attention that several companies may be making a profit by incorporating stuff I developed (available for free on this site) into their commercial products without even as much as asking permission. As such, to put an end to this, all the code you will find on this site is now distributed under the GNU General Public License v3 (GPLv3).

These scripts are based on GainData2fxt.mq4, an older script by an author that I couldn’t determine and on FXTHeader.mqh, written by stringo@codebase.mql4.com. The first one has suffered major modifications at my hand, while the second one was only modified here and there to adapt it to the updated FXT header structure.

You have to perform the following steps in order to convert the CSV data:

  • Copy the files from the zip archive to your MT4 installation. There’s a directory structure inside, make sure they land in the proper places.
  • Move the tick data file (the CSV file) to experts\files
  • Open a chart for the pair that you have data for (if you have EURUSD.csv you MUST open an EURUSD chart)
  • Select the timeframe that you wish to generate the FXT for (if you want to backtest on H1, then select H1 as the chart timeframe)
  • Double click the JForex2FXT script (or Dukascopy2FXT if you used the PHP method) in the navigation panel (it’s in the scripts section).
  • Configure the parameters in the window that pops up.
    For CsvFile you don’t need to write anything if the file is named just like the symbol and has a CSV extension (e.g. EURUSD.csv); otherwise, just type the file name there.
    Select “true” for CreateHst only if you want the script to create HST files. You do not need to perform this step more than once (leave it on “false” if you’re regenerating data for a different timeframe), because if this parameter is set to true, history will be generated for the whole span of the CSV file and for all timeframes, regardless of the start/end date you configure (note that the FXT file is still limited by those).
    By using the Spread parameter, you can configure the desired spread of the FXT file. If left at 0, it will use the current spread that your broker has. A special note here would be that many brokers are widening their spreads during the weekends, so pay close attention. If set to anything other than zero, it will be used as the spread. If you want a spread of 2 pips, enter 2.0, no matter if your broker has 4 or 5 digits. There’s also a small trick to change the spread without regenerating an FXT: open the existing FXT file with a hex editor; the spread is at offset 0xFC.
    The StartDate and EndDate fields control the span of the FXT file. MT4 has a problem with files larger than 2GB but the patch included in the archive fixes it, so it’s no longer mandatory to use these parameters. The format of the StartDate and EndDate fields is YYYY.MM.DD and if left blank the script will just use the earliest, respectively the latest date available in the CSV file.
    You can choose to use the real (variable) spread by enabling UseRealSpread. Note that the FXT files created with this parameter enabled will only work properly when the patch for real spread is applied. If using real spread, you can pad it by a given number of pips – if you want to pad it by 0.8 pips, just type 0.8 for SpreadPadding.
    If you want your resulted FXT to have commission, you can configure the PipsCommission value.
    As of 07.02.2011, upon Radulovic Attila’s suggestion, I added a Leverage parameter, set to 100 by default. Obviously, this changes the leverage of the resulting FXT file and the default value leads to a setting of 1:100. Radu has an MT4 programming blog and those interested will find there most of the backtesting info available on this page translated into Hungarian.
  • Click “Ok”. Once you do that, the data generation process will start and it will typically take 5 to 30 minutes, depending on the data range and volume, possibly even more if you’re using a slow machine. When the processing is finished, you will get an alert and some data will be printed in the Experts log.
  • Some people experienced miscellaneous issues. If your particular case is not listed here, feel free to drop a comment, I might be able to help. Also, take a look at the comments, your issue might have already been discussed.
    Among the common problems that the people encounter it’s worth mentioning the Windows 7 / Windows Vista users that run MT4 from their Program Files folder and that are unable to find the resulting files. The issue is caused by the fact that they have UAC enabled and these operating systems use folder virtualization. For Windows 7, the resulting files can be found in c:\ProgramData\ typically, while for Windows Vista they can be found in c:\Users\username\AppData\Local\VirtualStore\Program Files\. If they’re not there, just search for *.FXT, they must be somewhere. To get around this problem, either copy your MT4 folder in a location that’s not protected by UAC or simply disable UAC (type UAC in the start->search box thing and follow the on-screen process).
    Another issue experienced by some users is the lack of a MIN_LOT and LOT_STEP for the symbol in the resulted FXT file materialized in Order Send Error 131 when backtesting. This is caused by generating the FXT file with a MT4 client that wasn’t connected to the broker when the script was started. When generating an FXT file, your terminal should be connected to your broker.
    Last but not least, if your backtest stops immediately and you get an error “TestGenerator: internal error because the file not opened” in the backtesting journal, it means you forgot to apply the patch-script, but since your FXT is read-only, it was not overwritten. All the FXT scripts set the generated files read-only now, you will thank me for that at some point.

Copying the data to the correct location

After the script above has finished running, you will end up with a lot of files in your “experts\files” folder. At this point, exit the MT4 terminal then proceed to moving any .HST files from “experts\files” to “history\your_server_name”. Pay very close attention if you have multiple server directories in your “history” folder – you will have to move them into the one that’s correct for active account!
Note: unless you modify the CSV or get a new one, the HST files only need to be generated and moved once.
After moving the HST files, move the generated FXT file from “experts\files” to “tester\history”.

Using the data in a backtest

Since the recent MT4 clients no longer offer the ability to use custom FXT files, you need to run a special script that patches your MT4 client at runtime. It is included in the same archive linked above, but I’ll also link it here for the sake of convenience. It should work with builds 225-229, but it will probably work with previous builds as well and perhaps even with future builds.
So, to use the FXT files you created in a backtest, you have to install the scripts in the archive above in your MT4 installation, open any chart and run the “birt’s patch” script on it. By default, it will fix the 2GB limitation problem and it will also enable your MT4 to load FXT files. If you have FXT files created using real, variable spread you also need to enable the RealSpread parameter, but please note that once you did that, your backtests will ONLY work with variable spread FXT files and if you want to perform a backtest with “normal” spread you have to restart your MT4 terminal.
What the patch mainly does is disabling the MT4 default behavior to regenerate the backtest files every time you start a new test – so, if a file is already there, a new one will not be generated. By placing the FXT file you generated with the script in the tester\history folder, you ensure that it’s the file it is going to use for your backtest; of course, the selected backtest currency pair and timeframe must coincide with what you generated the FXT with.
Note that even though you can use files over 2GB, there is still a limit at 4GB. This can also be removed, but I see no need for it now since I modified all the FXT scripts to skip duplicate ticks which greatly reduces the size of the resulted files and consequentially the backtest duration. Just to be clear, to be considered duplicate, two ticks must match the following conditions: they must be consecutive, they must be within the same minute and finally they must have the same ask and the same bid price.

Special note: the 2GB patch will not work on Windows XP, Windows 2000 and Windows Server 2003 for MT4 builds prior to 405.

If you want to launch more than a single MT4 instance from the same folder (for concurrent backtesting with different parameters, for example), you can use one of my loaders, depending on which MT4 version you have. As a fun fact, Metaquotes went through the trouble of adding an in-memory CRC check on the terminal startup to prevent the creation of such loaders starting from build 228. I guess it wasn’t much help. I wonder why they did it. If you’re interested in the offsets, dig in the comments.

How to verify it worked?

That’s easy: if your test is running and the file in tester\history was not recreated, it usually means everything is ok. Naturally, it’ll skip the “Collecting M??” and “Using M??” steps.

GMT offset

This is a sensitive issue for many EAs. The Dukascopy data seems to be GMT-based all around the year, so the GMT offset that you need to use is 0. The FXT MQL scripts now have an external parameter that lets you specify the GMT offset you would like for the resulting data, both HST and FXT.

Troubleshooting

Note: The new loaders are only useful for starting multiple instances of MT from the same folder, they have no effect on being able to run a tick data backtest.

If your FXT files are being overwritten it means one of the following:

  1. You forgot to run the “birt’s patch” script. Note that it has to be run every time you want to use an FXT file you generated.
  2. DLL imports are not enabled (Tools->Options->Expert advisors).

If your FXT files are being recreated, check the following:

  1. You are backtesting on the same currency pair & timeframe as the one you created the FXT for.
  2. DLL imports are enabled.
  3. You ran the “birt’s patch” script and it reported some weird stuff in the experts log (one of the lines should say “FXT overwriting disabled.” among other things)

If your resulted FXT/HST files are very small in size, you are very likely using JForex to export and a locale that has a comma instead of a dot as the decimal separator. You can temporarily switch to the English locale (basically you have to go to Control Panel->Regional Settings and do it; use Google if you need more info) or you can try to start JForex in English, someone reported having had success with that.

If you can’t find any FXT/HST in your experts\files folder and the script reports successfully writing XXX bars/ticks, it’s probably caused by the fact that you’re using Vista or Windows 7 and you have UAC enabled. You should try looking for the files in C:\Users\[USERNAME]\AppData\Local\VirtualStore\Program Files\[TERMINAL]\experts\files\.

If your backtesting journal contains an error that says TestGenerator: internal error because the file not opened it means that your FXT file was correctly set read only and that you copied it to the correct location, but you did not run the birt’s patch script.

Gain Capital tick data

Gain Capital is the company behind forex.com. From what I hear, they’re a decent broker. They also offer some free tick data. If you’re feeling a bit masochistic and want to play with it, I recommend using a download manager to fetch it. Fair warning: the quality is pretty bad – not only is the data missing large chunks of time, but it’s also sporting misaligned periods (e.g. you can sometimes find data from the previous week in the file for the current week).
Still, before finding the awesome Dukascopy data, I used this for testing so I perfected some scripts for it, available here.
Inside you will find:

  • A script to process the data into a CSV. This script is heavily based on the original by carloss from the indo-investasi forum.
  • A script to somewhat filter & sort the resulted CSV.

Since these are totally obsolete to me, I’m not going to provide more details and you’re going to have to figure out how they work. Shouldn’t be too hard, though.

Oanda tick data

If you’re one of the lucky people who can get the Oanda tick data (it’s only available to people who have an account there funded with over 1k USD), get it in CSV format and use the Oanda2FXT MQL script provided in the scripts archive. It works just like the Dukascopy2FXT script described above.

The Oanda format seems to be varied, probably because it can be selected when requesting the data or something like that. I explained how to work around this problem in a comment on page 2, but given the large amount of comments on this article I don’t expect people to actually find it, so I made an alternate Oanda script (also included in the archive) that handles tick data files with values separated by a comma.

Variable spread backtesting

If you want your backtest to be even more accurate, you also have the option to use the real spread from the tick data. All you have to do is:

1. Make sure you’re using the latest FXT scripts.

2. When running the script, set UseRealSpread to true.

3. When running “birt’s patch”, enable the RealSpread parameter.

Paolo of siolifx.com ran the data integrity checker script from mt4i.com on Oanda and Dukascopy data and as it seems, although the Oanda data spans a larger time interval, it also has a lower quality. With his permission, I uploaded his Dukascopy EURUSD results and Oanda EURUSD results so everyone can take a look.

Ninjatrader export

A kind soul nicknamed xTrader1 has modified the process php script and made it export data that can be used in Ninjatrader. Quoting from his comment that is also available below if you’re into digging:

I tweaked your PHP script “process_dukascopy_data.php” in such a way, that it creates an original CSV file, but in addition 2 files Bid and Ask prices in ready to import NinjaTrader format.

He’s also been decent enough to send me the script and give me permission to host it here. Just as a disclaimer, I haven’t actually tested it so if it kills your dog or eats your babies I can’t be held responsible. Without further ado, I give link you the ninjatrader parse script. For some reason that I can’t explain, I got quite a kick out of naming the archive “dukascopy ninja script”.

Candle data

Starting from 11.02.2011, a pair of scripts that will allow you to download and process the Dukascopy candle data is available in the PHP scripts archive. Both of them work just like their tick data counterparts so no additional instructions are needed. They will download and process the M1 datal; if you want to change that, feel free to play with the source code. Since my time is very limited, they were only tested very briefly. If you run into any problems, please drop a comment or just use the contact form.

Short FAQ

Q: Why would I need this instead of tick data?
A: Because it’s available starting from 2003 while tick data is only available starting from 2007.

Q: Why would I use this instead of the Metaquotes data which starts from 1999?
A: Because the Metaquotes data has a lot of large gaps.

Q: Is there any way to import it in Metatrader and use it for backtesting?
A: Not really. Due to the volume data (expressed in traded currency), it’s not possible to make an accurate guess in regards to the number of ticks that occurred, which is what Metatrader needs. The solution would be to import the number of ticks from the Metaquotes history and randomly extrapolate where there are gaps, but currently there is no automated tool to do this.

Download links

For easier reference, here you go:
Dukascopy data download / parse scripts
MT4 terminal patch and FXT creation scripts
Multiple MT4 instance loaders
Gain Capital data tools
Tim’s DukasCopier
.NET Framework 4.0 (required for Tim’s DukasCopier)
Visual C 2010 runtime (x86) (required for removing the 2GB limit on Windows XP/2003 Server with MT4 builds 405+)
Visual C 2010 runtime (x64) (required for removing the 2GB limit on Windows XP x64/2003 Server x64 with MT4 builds 405+)

Show update history »

Update history

Since there are frequent updates to the scripts and I didn’t think to add some kind of versioning system, I decided to keep some kind of a changelog here, just to let you decide whether you need to update or not. The version linked above is always the latest.

  • 16.11.2011 – Fixed the crash in optimization and subsequent backtesting in builds 405 and above.
  • 11.09.2011 – The changes in builds 405 & 406 were major from the point of view of the tick data tools. As a consequence, a new generic loader was needed and this one works only with builds 405 and up; the old one is still in the “old” folder inside the archive in case anyone needs to use it on builds 402 and below. The MQL scripts were also updated to work with the new builds and as a side bonus, since I had to fully rethink the 2GB limit approach, I also made it run on Windows XP, but only for builds 405 and up and it requires the Visual C 2010 runtime; see above for the runtime download links.
  • 04.06.2011 – Added generic loader because the whole thing with Metaquotes releasing 4+ different builds with the same build number was getting on my nerves. This should work for builds 228 and up, if you need a loader for an older build look in the old directory in the archive. First time you run the generic loader, it will find the offsets; don’t panic and follow the instructions. If you update your MT4 build, it will just find the offsets again. If it fails to find the offsets for a new build just post a comment here.
  • 26.05.2011 – Added loader for yet another 401 build.
  • 15.05.2011 – Added loader for build 402.
  • 14.05.2011 – Added some more pairs to the PHP downloader script: USDSGD, AUDCAD, AUDCHF, CADCHF, EURNZD, GBPAUD, GBPCAD, GBPNZD, NZDCAD, NZDCHF, NZDJPY, XAGUSD and XAUUSD. The above-mentioned pairs are available starting from February 2010, with the exception of USDSGD (September 2008), XAGUSD (November 2010) and XAUUSD (May 2011). The batch scripts were also updated and the processing script was replaced with the workaround script because it seems that PHP bug is there to stay.
  • 14.05.2011 – The patch script is now able to remove the 2GB FXT limit for builds 399 and above. So far it works for 399 and 401 and hopefully it’ll work on future builds, too.
  • 14.05.2011 – Added loaders for build 399 and the two different 401 builds I ran into.
  • 07.04.2011 – Metaquotes has released a new build 392 with the same size, that resulted in a CRC error in the loader. I updated the loader and disabled the CRC check. Both 392 builds should work with the same loader now, but you need to update it if you have the old one and get the CRC error.
  • 26.03.2011 – Updated Dukascopier to v0.4, which contains some fixes. The issue with a comma as the decimal separator on some locales has been addressed.
  • 17.03.2011 – Added loader for build 392.
  • 14.03.2011 – Updated Dukascopier to v0.3.
  • 11.03.2011 – Added yet another loader for build 388. Metaquotes persists in its nasty habit.
  • 11.03.2011 – If the terminal is not connected, the FXT scripts will now wait for 15 seconds before exiting with an error message. This should allow them to be used with startup ini files.
  • 02.03.2011 – Metaquotes has acquired a nasty habit as of late. They release a build (in this case 388) and several days later they probably encounter some bug that must be fixed, address it and proceed to release a new build with the same number instead of increasing it. Just like 229, there are now two different 388 builds. I’ve added a loader for the new one, marked 388a. I also added detection for this build in the patch – you don’t need to upgrade, though; the old one will work just fine.
  • 01.03.2011 – Updated Tim’s DukasCopier to v0.2. This version adds candle data downloading.
  • 28.02.2011 – Once again, the old patch worked with the new 388 build, saying “unknown version” in the log. Now it is also able to identify 388. If you don’t have a problem with the patch saying “unknown version”, you don’t need to update.
  • 28.02.2011 – Added loader for build 388.
  • 27.02.2011 – Added Tim’s DukasCopier.
  • 24.02.2011 – Although the old patch works just fine with build 387, it would say “unknown version” in the log. That has been changed. There’s no need to update unless that message bothers you.
  • 24.02.2011 – Replaced short PHP open tag with full tag in download_dukascopy_candle_data.php
  • 24.02.2011 – Added loader for build 387.
  • 16.02.2011 – The margin currency used in the FXT files is now the account currency. Please note that this might not work properly if you hide some pairs. As a rule of the thumb, make sure that all currencies involved in the backtest have a “link” to the base account currency. For example, if your account currency is AUD and you’re backtesting EURJPY, ensure that both EURAUD and AUDJPY are among your visible symbols.
  • 15.02.2011 – In addition to setting the GMT offset of the resulting FXT, you can now set the DST according to the US or European legislation.
  • 13.02.2011 – Fixed two bugs that were leading to an incorrect commission calculation.
  • 11.02.2011 – The MQL scripts now perform a NormalizeDouble(price, Digits) on the data read from the CSV files, meaning that the FXT files generated and used on 4 digit brokers will no longer result in invalid price errors.
  • 11.02.2011 – A scripts that allows processing Dukascopy candle data has been added. A script for downloading it was already in there.
  • 09.02.2011 – The FXT scripts will print a message in the experts log when starting to process a new year of data. The readonly attribute is now set at the very end to allow printing & alerting even if DLL calls are disabled.
  • 07.02.2011 – Added alternate PHP process script for people running into CLI crashes.
  • 07.02.2011 – Added Leverage option to all FXT processing scripts.
  • 19.09.2010 – Updated the FXT scripts to also write the first and last tick of the file as its start/end date, thus avoiding the 1970 dates displayed in the results.
  • 21.04.2010 – Updated the process script to resume if the output file exists, effectively allowing incremental CSV generation.
  • 16.11.2009 – Implemented the fix Futron posted in the comments section.
  • #1791 written by fxtech January 15, 2012 (1 week ago)

    Is any one else having trouble logging in to jforex besides myself and Eddie? I just opened a new demo account this weekend, but it says “authentication failed”. Could it be because it is the weekend?

    • #1792 written by birt January 15, 2012 (1 week ago)

      Dukascopy might be having some issues, the tick data files are all 404 as well. Perhaps a server migration of some sort.

  • #1793 written by Mat January 15, 2012 (1 week ago)

    Same problem here for jforex.
    However running ducascopier without the convert in csv option is very very fast, the there will be the new jforex2fxt script it will be working all perfectly…..

  • #1794 written by Mat January 15, 2012 (1 week ago)

    By the way, now I am processing the euraud data with php to create the csv file.
    Sometimes I get some warning about. Issuing data file in some folder…. It is not a Saturday or a Sunday …. So are those error in the tick data? Is this normal? Or should I use the php script to grab them course is never failing?

    • #1795 written by birt January 15, 2012 (1 week ago)

      It’s probably nothing you should be worried about but I can’t tell you anything else without the exact error messages.

    • #1796 written by birt January 15, 2012 (1 week ago)

      Also, if you continue posting comments here please either correct your email address or otherwise stop clicking the “notify me” box when using an invalid email address.

  • #1797 written by trader January 16, 2012 (1 week ago)

    May i know whether i can optimize my ea using the tick data?

    • #1798 written by birt January 16, 2012 (1 week ago)

      The patch script might work with optimization with builds 402 or below. Mind you, there’s no guarantee it will and you might run into other problems such as the EAs not working with those builds.

      This month I will release a commercial tool that will allow optimization with tick data with all builds.

      • #1799 written by Alex January 22, 2012 (5 days ago)

        Hi Birt, will the commercial tool also enable us to use the Work Forward analyser (disabling overwriting permanently, so the script doesn’t need to run at the beginning – I guess this is causing the incompatibility at the moment). Thanks

      • #1800 written by Alex January 22, 2012 (5 days ago)

        Ah and also, will we be able to use multiple instances with the Walk Forward Analyzer (which starts up MT4 via command line)? Multiple instances really speed up my testing..

        • #1801 written by birt January 22, 2012 (4 days ago)

          The current beta already works with the Walk Forward Analyzer. The existing files don’t get overwritten, but you have to make sure that your FXTs have the whole range required by the WFA.

          As for starting multiple instances, I don’t know. If the WFA supports that, then yes, definitely. Running the loader repeatedly allows running multiple instances from the same folder.

  • #1802 written by trader January 16, 2012 (1 week ago)

    My metatrader always restarted everytime i press the start button on strategy tester. So i can’t backtest. Do you have the solution?

    • #1803 written by pipsaw January 16, 2012 (1 week ago)

      I experienced the same issue couple of times. Try applying the Birt’s patch as first thing after starting MT4 terminal. That worked for me!

  • #1804 written by Tyler January 16, 2012 (1 week ago)

    Dukascopier is still reporting nothing but 404 errors trying to find data on Dukascopy’s website. Exmaple:

    D:\Forex\DukasCopier\EURUSD\201201\21h_ticks.bin – The remote server returned an error: (404) Not Found.
    D:\Forex\DukasCopier\EURUSD\201201\22h_ticks.bin – The remote server returned an error: (404) Not Found.

    Is anyone else having any luck?

    • #1805 written by pipsaw January 16, 2012 (1 week ago)

      Sorry, forgot to hit the reply button earlier –

      Tyler I dont think the issue is with Dukascopier, no luck on php or Dukascopy website either. Interestingly, the data is gone from the start of 2008. This is what I have from php script (i can’t get anything after this timestamp) –

      Info: Downloading EURUSD starting with
      WARNING: missing file http://www.dukascopy.com/datafeed/EURUSD/2008/00/01/00h_ti
      cks.bin (1199145600)

  • #1806 written by pipsaw January 16, 2012 (1 week ago)

    Tyler I dont think the issue is with Dukascopier, no luck on php or Dukascopy website. Interestingly, the data is gone from the start of 2008. This is what I have from php script (i can’t get anything after this timestamp) –

    Info: Downloading EURUSD starting with
    WARNING: missing file http://www.dukascopy.com/datafeed/EURUSD/2008/00/01/00h_ti
    cks.bin (1199145600)

    • #1807 written by Tyler January 16, 2012 (1 week ago)

      Thanks for the confirmation, pipsaw. It’s been two days already…hopefully nothing permanent!

      • #1808 written by birt January 16, 2012 (1 week ago)

        Yeah, it’s been the weekend. They’re hopefully back to the office and already working on fixing the issue. As pipsaw mentioned, only data for the past 3 years is missing and the history downloader on the Dukascopy website is not working either, which probably means the whole thing is just a fluke, nothing permanent. Have some patience, I’m pretty sure it’ll come back up.

        • #1809 written by Tyler January 17, 2012 (1 week ago)

          Just to update everyone on the data available issue, while most pairs still seem to be down, I was able to get data for AUDNZD, EURAUD, EURUSD, GBPUSD, and GBPJPY last night. Hopefully the other pairs will be available again soon.

  • #1810 written by Mike January 16, 2012 (1 week ago)

    I just started the php process and after not too long I started getting a lot of missing file messages. I tried accessing the .bin file directory from a different network connection to see if maybe they blocked my IP or something along those lines, but I’m getting 404 errors from there too, even when I try accessing one of the files I was able to successfully download when I first started the process. It looks like they may have removed the datafeed. Can anyone confirm? Is this data not available in a zipped format online somewhere or would that be some sort of violation???

  • #1811 written by Mike January 16, 2012 (1 week ago)

    and of course I would read the comments above mine AFTER I posted it lol

  • #1812 written by Szilard January 17, 2012 (1 week ago)

    Hi there. Is that normal that if “use real spread (11)” and “use fix spread (20)” are appied to a M5 backtest after each other, then the results vary dramatically?
    I ran an EA on USDCAD 2011.01. fix 0.1 lots 1000EUR starting equity.
    The real spread version resulted +123 EUR in one month
    The 2 pip spread version resulted – 155 EUR in the very same month.
    Strange thing is that even trade openings differ in the two versions. The EA uses price levels as trade open condition, not indicators.

    Additionally strange thing is that in the real spread version no volumes indicator can be calculated. Volumes is 0 continously.
    The condition “if (iVolume(NULL,0,0) ==1 ) ” never gives true result.

    What could cause so big difference?

    • #1813 written by birt January 17, 2012 (1 week ago)

      If your EA uses the volume in any way, it can’t be tested with real spread because when real spread is enabled, the spread replaces the volume data.

      • #1814 written by Szilard January 17, 2012 (1 week ago)

        Wow, thanks, i did not know that. However this might not be the reason for the big difference between the two tests.

        • #1815 written by birt January 17, 2012 (1 week ago)

          Yeah, it’s probably what caused the difference. On the other hand, if your EA had better results with volume = 0, you might want to think about taking it out of the picture.

  • #1816 written by Meena January 18, 2012 (1 week ago)

    Have an account with Oanda so I have access to their tick data. Run Birt’s patch set properly. I have a file for EURUSD I have it named: eurusd 11-1-10 to 11-1-11. It is placed in Experts/file. I have followed the instructions but I get the message “cannot open input file”. mt4 build 409. Have tried both Oanda files… any idea what’s going on? the fxt file is in include folder..
    thanks…

    • #1817 written by birt January 18, 2012 (1 week ago)

      If it says “cannot open input file”, it means one of the following:

      • Your file was not named as the pair name (EURUSD.csv) and you did not type the file name in the input parameter.
      • You forgot to type the file extension (.csv, .txt or whatever it is) when typing the file name in the input parameter. Please note that Windows Explorer might be configured to hide extensions for known file types which might be confusing you into thinking it has no extension.
      • If the above do not apply, you did not copy the file to the correct location
  • #1818 written by Alex January 18, 2012 (1 week ago)

    Please, tell me how I can download only eur|usd data with “download_dukascopy_data.php” as I don’t need others like aud|jpy with which the downloading begins ?

    • #1819 written by Simon D January 18, 2012 (1 week ago)

      you need to comment out the other currencies at the beginning of the program

      • #1820 written by Alex January 18, 2012 (1 week ago)

        What program should I use to edit *.php file ?
        Should I use “;” in front of other currencies to comment a line ?

        • #1821 written by Simon D January 18, 2012 (1 week ago)

          use notepad or wordpad or any other text editor. You need to comment out like this:

          //”EURSEK” => 1175270400, // starting from 2007.03.30 16:00
          “EURUSD” => 1175270400, // starting from 2007.03.30 16:00
          //”GBPCHF” => 1175270400, // starting from 2007.03.30 16:00

          every currency apart from the one you want two slashes in front

  • #1822 written by Alex January 18, 2012 (1 week ago)

    Ok, thanks. I just deleted all other currencies, now eur|usd is downloading.

  • #1823 written by Meena January 18, 2012 (1 week ago)

    address: C:\Program Files (x86)\OANDA – MetaTrader\experts\files

    file name: eurusd 11-1-10 to 11-1-11.csv

    Window 7

    Am sure I have the files placed and named correctly.
    If it is explorer blocking csv how do I fix that?
    thanks…

    • #1824 written by birt January 18, 2012 (1 week ago)

      If you’re using Vista, search the comments for a solution.

      If not, I would recommend renaming the file to EURUSD.CSV and trying that way. You might be mistyping its name or something like that.

  • #1825 written by Meena January 18, 2012 (1 week ago)

    Renaming the file does not work.
    In the past I have run Dukascopy successfully but it was in 2010.
    I have an old setup for FXDD from 2010. I’m going to try to install it without updating it and see if that will work with Oanda data.
    Maybe it is mt4 build 409 that is the problem.
    Trying to run this on Windows 7.
    I am sure I am doing everything as instructed.
    I can also try to run this on Vista on another computer.
    Thank for your patience..

    • #1826 written by birt January 18, 2012 (1 week ago)

      Don’t bother trying with Vista.

      Anyway, the problem might be caused by the Windows UAC. I suggest making a copy of your MT4 folder somewhere else and trying from there – for instance you could put the whole folder on the desktop, there are no such issues there.

  • #1827 written by Meena January 18, 2012 (1 week ago)

    Yikes. Sure would like to know what the problem is.
    Folder is moved to desk top.
    csv file is named EURUSD.CSV and is in experts/file folder
    setup FXDD build 228.
    Ran Birt’s patch on EURUSD
    tried both Oanda scripts
    still have message cannot open file?
    windows 7.

  • #1830 written by Meena January 18, 2012 (1 week ago)

    yes it is in experts/file.
    not experts/files

    I am trying to Google for discussion about similar problems…
    If I try to open the CSV file with notebook it says stopped working, would that just be because is it a large file 471020KB and it will take some time for it to open in notebook? Is it too big for notebook to open?
    The file in question is Oanda separated by commas.

    • #1831 written by Harald January 18, 2012 (1 week ago)

      you need special editors to open such large files. but this is not your problem.
      you wrote:
      yes it is in experts/file.
      not experts/files

      That’s WRONG! It must be in expert/files

      • #1832 written by birt January 18, 2012 (1 week ago)

        Yes, Harald is 100% correct.

  • #1833 written by Meena January 18, 2012 (1 week ago)

    notebook gives message (not responding) not stopped working.

  • #1834 written by Harald January 18, 2012 (1 week ago)

    Hi birt,
    at the next opportunity, can you fix a small issue in the php scripts. Windows-php does not support several date/time modifiers, like %D and %T, giving a warning.
    > gmstrftime(“%D %T”,$firsttick)
    you can replace it by
    > gmstrftime(“%Y-%m-%d %H:%M:%S”,$firsttick)
    if you like.

  • #1835 written by Meena January 18, 2012 (1 week ago)

    The csv file is in the proper folder. My message on the thread has a wpelling errors. sorry about that But the csv file is in the proper folder. That is not the porblem.
    I have doenloaded the Process Manager. And filtered for terminal.exe.
    How to find the log?

  • #1836 written by Jason January 18, 2012 (1 week ago)

    Birt

    I seem to be having problems when running optimisation after running the Brits Patch. I just wanted to check the following in the Experts journal is correct:

    2012.01.18 22:32:18 birt’s patch EURUSD,M15: removed
    2012.01.18 22:32:18 birt’s patch EURUSD,M15: uninit reason 0
    2012.01.18 22:32:18 birt’s patch EURUSD,M15: Process patched for variable spread at 0×557016. Volume check removed at 0x55734E.
    2012.01.18 22:32:17 birt’s patch EURUSD,M15: Patched: fseek 0×557244, fopen 0×557154, fclose 0x55710E, fclose 0x555BD5, fclose 0x5563EE, fread 0x55729A, filelength 0x55718D. 2gb limit removal patch was installed at 0×4270000.
    2012.01.18 22:32:17 birt’s patch EURUSD,M15: FXT overwriting disabled. Addresses patched: 0×556122, 0x55623B, 0×556343.
    2012.01.18 22:32:16 birt’s patch EURUSD,M15: MT4 build 409 detected.
    2012.01.18 22:32:16 birt’s patch EURUSD,M15 inputs: NoFXTOverwrite=true; Remove2GBLimit=true; WARNING2=”Using variable spread with a non-variable spread FXT does NOT work.”; RealSpread=true;
    2012.01.18 22:32:06 birt’s patch EURUSD,M15: loaded successfully..

    Jason

    • #1837 written by birt January 19, 2012 (1 week ago)

      Optimization does not work with that script with build 405+ and it has problems even with previous builds.

      I am working on a loader that also supports optimization, but it’s going to be launched as a commercial product. It’s already been in beta testing for a while. Keep a close eye on the site during the following couple of weeks if you are interested.

  • #1838 written by Meena January 19, 2012 (1 week ago)

    I tried a couple of things which seem to have fixed the problem of “cannot open the file” in windows 7 for Oanda build 409. In permission I disabled “WMPNetworkSvc”. Named the file simply EURUSD, although I had tried to run the script before with that name with no success. Anyway the file is converting now. Thank you…

  • #1839 written by Meena January 19, 2012 (1 week ago)

    I have the file converted to hst and they in the proper folder The fxt is in its proper file. I opened the terminal ran Birt’s patch, then built on the EURUSD chart. As soon as tried to run the tester the platform closed.

    • #1840 written by birt January 19, 2012 (1 week ago)

      That can be caused by a number of things:

      • You’re running a recent EA on builds 402 or earlier
      • You have run another backtest prior to running the Birt’s patch script
      • You’re trying to run an optimization.

      Judging by what you wrote above, you are running version 228 which will crash with pretty much every other EA available nowadays. I suggest trying to attach the EA to a chart before trying to backtest it. If your terminal build is 402 or below, consider trying with build 409.

      Other options to try: restart the terminal, run the birt’s patch script as soon as it starts, attach the MACD EA to a chart and if that works, backtest it.

      As for your previous problem with the CSV file, it was most likely caused by the fact that you named it EURUSD.CSV.CSV but windows explorer was hiding the file extension. Google “hide extension for known file types” if you wish to disable that.

  • #1841 written by Alex January 19, 2012 (1 week ago)

    I reloaded tick data with php, again followed all the instructions so everything is in its place, run the patch script; but when I try MACD_Sample or any other (it skips the “Collecting M??” and “Using M??” steps) I get 25% modeling quality. I decided to check the data with the “MT4i Data Integriry Checker”, it says that data is OK. How can I finally get 99% ? Help.

    • #1842 written by birt January 19, 2012 (1 week ago)

      I would venture a guess that you are using the “control points model” instead of “every tick”.

      If that’s not it, check that you have DLL calls enabled (the FXT should be impossible to overwrite because it should be set as read-only if you had that enabled when creating it) and if you do, please post the lines related to the birt’s patch script that appear in your experts log after running it.

      • #1843 written by Alex January 21, 2012 (6 days ago)

        I use “every tick”.
        Here is the log:
        “”
        13:16:33 birt’s patch GBPUSD,H1: loaded successfully
        13:16:43 birt’s patch GBPUSD,H1 inputs: NoFXTOverwrite=true; Remove2GBLimit=true; WARNING2=”Using variable spread with a non-variable spread FXT does NOT work.”; RealSpread=false;
        13:16:43 birt’s patch GBPUSD,H1: MT4 build 409 detected.
        13:16:43 birt’s patch GBPUSD,H1: FXT overwriting disabled. Addresses patched: 0x5560E2, 0x5561FB, 0×556303.
        13:16:45 birt’s patch GBPUSD,H1: Patched: fseek 0×557204, fopen 0×557114, fclose 0x5570CE, fclose 0x555B95, fclose 0x5563AE, fread 0x55725A, filelength 0x55714D. 2gb limit removal patch was installed at 0×3840000.
        13:16:45 birt’s patch GBPUSD,H1: uninit reason 0
        13:16:45 birt’s patch GBPUSD,H1: removed
        “”

        • #1844 written by birt January 22, 2012 (4 days ago)

          If you still get 25% check that you actually generated the FXT for the timeframe you are testing. Could be that you’re backtesting using M1 and the FXT was generated for another timeframe.

          • #1845 written by Alex January 23, 2012 (4 days ago)

            Hi, Birt.
            I am actually using m1 and the FXT file is generated for m1 also. I am confused, don’t know what to try now.

          • #1846 written by birt January 23, 2012 (4 days ago)

            I’m not sure what could be wrong but getting a 25% modeling quality on M1 means one of the following:
            - you are using control points
            - you are using regular backtesting (interpolated ticks)

            This means the FXT you generated was not used, either because it wasn’t copied to the correct location or because it was overwritten when you started the backtest. Judging by the log you posted you seem to have correctly run the script (I’m assuming prior to the backtest), so I would check that the tick data FXT is in tester/history. Copy it once more if you have to because if you got 25% once, you must have a 25% percent FXT in there. Another thing to make sure is that you’re copying it to the correct MT4 folder if you have multiple installations on your computer.

  • #1847 written by Jason January 19, 2012 (1 week ago)

    Birt

    Is there a Script that works with Build 409 that I can download?

    I’m also up for any beta testing ;-)

    Jason

    • #1848 written by birt January 19, 2012 (1 week ago)

      The script that is available for download for about 2 months now works with build 409 with a few limitations (it must be started prior to any other backtest and optimization does not work).

      As for beta testing, one of the requirements is prior experience and a certain degree of familiarity with tick data backtesting. If you feel you meet this criteria, please get in touch with me using the contact form.

  • #1849 written by John Bennett January 20, 2012 (6 days ago)

    Bert – Awesome blog! The Dukascopy loader is better than a twelve pack, and maybe NFL football (go Giants!). Thanks, and I will check in often.

    John, Currency and equity trader

  • #1851 written by Tyler January 21, 2012 (6 days ago)

    I’m still getting 404 errors on most pairs for recent tick data. It’s been over one week now. Can anyone confirm they are seeing the same thing? If anyone is a Dukascopy client, do you know if Dukascopy has said anything about this? I appreciate any feedback anyone may have. Thanks.

    • #1852 written by birt January 21, 2012 (6 days ago)

      I have the same issue. I heard no news from Dukascopy but I didn’t even try to keep an eye on their channels. I suppose it’s going to be restored at some point, they may have had some problems.

    • #1853 written by Alex January 22, 2012 (5 days ago)

      Just to contribute something as well (even if little): I wrote an email to Dukascopy and they confirmed that the download issue is just a temporary thing and they are working on getting it fixed.. so it should be up again soon.

      • #1854 written by birt January 22, 2012 (5 days ago)

        Good to know, thanks! :)

      • #1855 written by Tyler January 22, 2012 (5 days ago)

        Yes, thanks for confirming, Alex. I appreciate it.

  • #1856 written by Jak January 21, 2012 (6 days ago)

    what do i type.. jsut to download EURUSD… with tht php method

    php download_dukascopy_data.php

    or lets say only EURUSD and USDJPY

    Thanks a lot

    Jak

    • #1857 written by birt January 21, 2012 (6 days ago)

      Please scroll up to comment #1815 just above and read that thread.

  • #1858 written by Jason January 22, 2012 (4 days ago)

    Does anyone have a build of MT4 that works with the Optimisation Script?

    I can only find Build 409…I have build but I’m not sure if it needs to connect to the broker or can you use it with the converted tick data without being connected.

    • #1859 written by Alex January 22, 2012 (4 days ago)

      209 works fine. I just downloaded it again, google will find it in some forum post..

  • #1860 written by Mike C. January 23, 2012 (4 days ago)

    Hey Birt, what hex editor did you use to edit the .fxt file? Is there an easy way to modify the leverage like there is the spread? Thanks,

    Mike

    • #1861 written by birt January 23, 2012 (4 days ago)

      Personally, I use hiew because it can also disassemble stuff but I wouldn’t recommend that one to you because it’d feel clumsy. Just get any windows hex editor, e.g. winhex.

      You can edit the leverage at offset 0×164 I think (or somewhere close to that, it should be 0×64 if you set it to 1:100, just look for that and change it).

      • #1862 written by Mike January 26, 2012 (20 hours ago)

        Awesome, I found the field and the corresponding value. I had actually set mine to 200, so it was C8. I’d like to change it to 300, but 300 in hex uses three digits…should it overlap into the field on the left or the right of C8? If you’re not sure, no big deal, just trying to see if I can save time from generating a new FXT file with different leverage if I don’t have to. Thank you!
        Oh by the way, found an amazing hex editor called Neo Hex Editor, looks up the whole 3gb file in an instant and is very browsing friendly

        • #1863 written by birt January 27, 2012 (7 hours ago)

          Into the field on the right. It should look like 2C 01.

  • #1864 written by Jason January 23, 2012 (4 days ago)

    Alex

    I’m assuming that if your using Tick Data there is no need to be connected to a broker first before running any optimisation?

    Jason

    • #1865 written by Alex January 23, 2012 (4 days ago)

      Well to convert the data via the JForex2FXT script you need to be connected. But I guess a demo account should be sufficient. The 209 build I found was customized for Sunbird, but I just copied my Alpari setting into it.. Later on you don’t need to be connected anymore. Do you know any security issues with this build?

  • #1866 written by SvenGali January 23, 2012 (4 days ago)

    Hi Birt,

    getting data from Dukascopy works really fine – great blog / great work – thx! I have an issue with JForex2FXT (FXPrimus Metatrader Build 409). The EUR.USD csv-file looks good (complete – checked with ultraedit), but when I apply the script to the 1M-chart, it only reads data until mid of 2010 (exactly 2010.06.09 19:36), then starts shortly with 2012 (but only weird data, i.e. 2012.12.31…) and then stops. If I load the according hst-files, I can see that most of 2010 and 2011 is missing and I think it’ll be the same with the fxt-file. I already tried to use a smaller time-frame but that doesn’t solve the problem. The file-size of the csv-file is 5.8 GB. Could you tell me more ’bout the possible cause? Thx in advance. SvenGali

    • #1867 written by birt January 23, 2012 (4 days ago)

      There’s an error in your CSV file. I’ve already written a script that is able to handle such cases over a week ago, but it changes things dramatically and I have to rewrite that part of the guide, plus the downloading part which I’ve already rewritten but not published yet. In other words, the script is ready but it is waiting for me to have the time to rewrite the guide, which might happen this week or perhaps the next.

      In the meantime, I suggest reading the last couple of comment pages for solutions as you’re not the only one that ran into this problem.

  • #1868 written by Jason January 24, 2012 (3 days ago)

    Alex / Birt

    I’ve managed to find a Sunbird build 229 and got it to connect to the broker using the login from a version 409. Converted the CSV and transfered the files but still no luck getting the MT4 to run a Optimisation.

    I’m converting the FXT using a 2.0 pip spread.

    Any suggestions?

    I’m running the MT4 on Windows 7

    Jason

    • #1869 written by birt January 24, 2012 (3 days ago)

      no luck getting the MT4 to run a Optimisation.

      That’s not very descriptive. How does it behave? Does it crash? If so, did you try your EA on a chart in build 229 prior to trying it in an optimization?

    • #1870 written by Alex January 24, 2012 (3 days ago)

      I use XP and it all works fine.

  • #1871 written by Jason January 24, 2012 (2 days ago)

    Birt

    I think my issue is with the tick data once converted…something is going wrong. I’ve tried both JForex & Dukas Copier CSV files and converted with the corresponding scripts to product both FXT & HST files then copied to the directories. I then run the Birts script and start the Optimisation, but I’m not getting any Opimisation results – i’ve tried several EA’s including the sample MACD sample. When i convert the CSV files shoulds i input both the Spread which I’ve been setting to 2.0 and also the Real Spread? or just select one or the other? I also found the that the newer MQL script makes the 229 build crash, so i’ve used an older Script I found in a post from last year…

    I’m just converting 1 year of tick data at a time, and tried leaving the Use date box ticked and unticked..

    Jason

  • #1872 written by Jason January 25, 2012 (2 days ago)

    Birt

    I forgot to add that i’m converting 1 year of data and I can get the MT4 to run only on 1 month of data, and it goves me optimisation results, any longer and it the Journal is saying : 2012.01.25 01:03:25 There were 4 passes done during optimization, 4 results have been discarded as insignificant…..

    Jason

    • #1873 written by birt January 25, 2012 (2 days ago)

      Discarded results means the runs were not profitable. You can disable discarding results by right clicking the optimization results window and enabling an option in that menu.

  • #1874 written by Jason January 25, 2012 (2 days ago)

    Birt

    My back tests run with 99% modeling quality and use the full years data, but only 1 month when optimising

    Jason

    • #1875 written by birt January 25, 2012 (2 days ago)

      Optimization is not supported with the Birt’s patch script. If you can get it to run, good for you; otherwise, if you need to run optimizations you can just wait for the Tick Data Suite which is now in beta testing and which should be released within the following couple of weeks.

      Basically, the script was never supposed to work with optimization although it seems to occasionally do so with builds older than 402 but with severe limitations (such as only selected periods). Please read the previous comment pages if you need any more information.

  • #1876 written by jore January 25, 2012 (2 days ago)

    Hi birt,

    I got the following messages when create fxt files with Dukascopy2Fxt.
    I create csv-files with dukasCopier.

    2012.01.25 10:01:25 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2029.
    2012.01.25 10:01:22 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2027.
    2012.01.25 10:01:21 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2012.
    2012.01.25 09:56:48 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2009.
    2012.01.25 09:34:15 Dukascopy2FXT EURUSD,H1: Starting to process EURUSD 2008.

    But no such years are in the csv-file, I checked it with uedit.
    The fxt file is relativly small (1,454,540kB), when I try it, MT4 crashes.

    I use MT4.209 Dukascopy2fxt from 11.03.2011.

    In the log of DukasCopier there are some “Zero length file skipped.” messages, but I think that is OK.
    And it finished with:
    EURUSD\20120\23\23h_ticks.bin – Der Remoteserver hat einen Fehler zurückgegeben: (404) Nicht gefunden.

    Weeks ago all things work well.
    Any ideas?

    Thanks

    • #1877 written by birt January 25, 2012 (2 days ago)

      It’s an issue with the Dukascopier. Please read the last couple of comments pages for more info.

    • #1878 written by Xman January 27, 2012 (14 hours ago)

      I had the same problem with incorrect years like 2029, 2031 showing up in the FXT only.

      Here’s a workaround :
      In my case, the issue seems to happen only when my csv files were over 4gb. So I used the “gsplit” freeware tool ( http://www.gdgsoft.com/gsplit/ ) to split my CSV into 2 files.

      My EURUSD 2007-2011 CSV filesize was 6.8gb and with gsplit configured to split the csv file after 67million linefeeds characters, it did the job nicely. It resulted in 2 files, first file had data from 2007/04 to 2008/11, second file had 2008/12 – 20011/06. I took a peak at the first few lines of the csv by using BeyondCompare and clicking stop before it loads the entire file and max out your ram. If there is a better text editor to stream view these massive files, I would love to know.

      Renaming the second file to EURUSD.CSV and re-running Dukascopy2fxt did the trick. No more invalid 2029,2031 years in the log. The caveat is you can only load one of the csv files into the FXT, I loaded the second file (2008/11 to 2011/06), the first file can be loaded separately (replacing current FXT) if needed later.

      Similar process worked on a Jforex file 2007-2012, split it so last file is under 4gb and run the converter. Its a workaround until MT4 or coverter can support over 4gb csv files.

  • #1879 written by Jason January 25, 2012 (2 days ago)

    Birt

    Looking forward to the Tick Data Suite…

    Keep up the great work..

    Much appreciated

    Jason

    • #1880 written by birt January 25, 2012 (1 day ago)

      If you’re the Jason P. who contacted me on the 21st, please contact me from a different email address as I haven’t been able to reply to the one you provided.

  • #1881 written by nicco January 25, 2012 (2 days ago)

    I can confirm Ticks have stopped download on JAN 23 @ 12h

    Missing Files from then on…

    Birt and ideas/New from Dukas?

    • #1882 written by Harald January 25, 2012 (1 day ago)

      it is Jan 23 2012, right?
      this is unrelated to the current problem that long periods of some currencies are missing.
      In Dukas data quite often the last 2-3 days are missing, but will be filled it a couple of days later.

  • #1883 written by Delbru January 25, 2012 (1 day ago)

    Hi Birt,

    For you info I want to back test my EA on tick data from dukascopy using the ‘JForex2FXT’. Data range is 09/01/2011 to 12/30/2011. I followed all the steps as you mentioned.
    When I ran the ‘birt’s patch’ from scripts I got the following error.

    2GB limit removal not activated.

    If you’re using Windows XP, consider getting a copy of the Visual C 2010 runtime, available at http://www.microsoft.com/download/en/details.aspx?id=5555 (x86) and http://www.microsoft.com/download/en/details.aspx?id=14632 (x64).

    Could not find the _fseeki64() function in your msvcrt.dll or msvcr100.dll!

    Also in the Forex.com Metatrader strategy tester I get the error for 5m and any other time frame.
    TestGenerator: no history data ‘XAUUSDFXF5′

    • #1884 written by birt January 25, 2012 (1 day ago)

      If you only wish to backtest such a short time period, you don’t even need to care about the 2GB limitation. The FXT files will be larger than 2GB only if they span several years.

      As for fixing the 2GB problem, if you’re using MT4 build 405 or higher, all you have to do is what it tells you: download the MSVC10 runtime.

      Finally, regarding the “no history data” message, that means that your history center does not have history for the XAUUSDFXF5 pair for the timeframe you selected. Try opening a chart with that pair and timeframe and keep pageup pressed for 30-60 seconds.

      • #1885 written by Delbru January 27, 2012 (18 hours ago)

        Hi Birt,

        I Downloaded 3 months worth of XAUUSD Tick data from 09/01/2011 to 12/30/2011.
        I placed the XAUUSDFXF in Forex.com MT4 ‘experts/files’ folder. I am not sure how to view chart of this particular dataset. Whenever I drag and drop the XAUUSDFXF from ‘navigator’ I only see the most current charts. My EA is sort of medium frequency algo which reads incoming tick flows and tries to predict price for a future horizons. The candles 5m, 15m, are only used for the EA to be able to restart itself everytime…. and also also for some post trade order management. My ea doesn’t use O,H,LC values of any candle either. My objective is to carry out an in-sample test my EA on this 3 months tick data. How do I do that?

        • #1886 written by birt January 27, 2012 (7 hours ago)

          If you wish to backtest it, all you have to do is follow the guide above. Basically, you have to convert your XAUUSD CSV data to an FXT + HST files, copy them to the correct locations and proceed with the backtest. If you want to view the resulting chart, just open a chart after you copied the HST files.

  • #1887 written by Neo January 26, 2012 (1 day ago)

    Birt, will your new Tick Data Suite allow for “Open prices only” Model back testing? It appears that currently the “Open prices only” model crashes MT4 whether running a single scan, or trying with Optimization.

    Thanks,
    Neo

    • #1888 written by Harald January 26, 2012 (1 day ago)

      Neo,
      just for curiosity: MT4 supports open-price backtesting, and with this method I feel “real” tick data is pretty useless. So what do you miss from what the MT4 platform is offering by default?

      • #1889 written by Neo January 27, 2012 (10 hours ago)

        The only reason I ask is because the WFA has to do a complete Optimization back test before doing its walk forward tests…so they suggest (to speed up the first optimization back test), that you use the Open Prices only for that leg of the journey.

        Neo

    • #1890 written by birt January 26, 2012 (1 day ago)

      Harald is correct, you don’t really need the patch or tick data if you want to use control points or open prices only. The CSV parsing script does not produce an FXT that can be used with these models.

      However, using the Tick Data Suite will no longer crash so in that respect I guess you could say it works.

  • #1891 written by Pawel January 26, 2012 (1 day ago)

    GMT offset for tick data from dukascopy is 0 , what is UTC offset for this data?

    • #1892 written by birt January 27, 2012 (8 hours ago)

      GMT is pretty much the same as UTC, so the UTC offset is 0. Google GMT vs UTC if you need more info on the topic.

      If you are asking if the data is shifted by 1 hour during the summer, it isn’t.

  • #1893 written by www.fxfir.com January 27, 2012 (12 hours ago)

    hello
    do you have direct download mt4?
    very thank you

  • #1895 written by Neo January 27, 2012 (10 hours ago)

    I don’t do Open Price only back tests, but the WFA needs to do a complete Optimization back test first, before doing its analysis…so in order to speed up the first scan, they suggest doing Open Prices only.

    Neo

    • #1896 written by birt January 27, 2012 (8 hours ago)

      The Tick Data Suite will work fine with all backtesting models, although FXT overwriting is only disabled for every tick backtests.

  • #1897 written by Jane January 27, 2012 (9 hours ago)

    Hi Birt,
    I can backtest my EA with Dukascopy Tick Data now, millions thanks !!!
    Another question please, you mentioned user can run MT4 EA on platform of JForex with a very tricky way, can I know what’s that? Because I really want to find a stable ECN like Dukascopy, thank you!!!

  • #1899 written by KlamseN January 27, 2012 (3 minutes ago)

    Hi birt, any idea why the 2gb removal doesn’t work for my new laptop?
    It’s a Win7 64bit maschine with 8gb ram.

    2012.01.27 19:39:33 birt’s patch EURUSD,H1: uninit reason 0
    2012.01.27 19:39:33 birt’s patch EURUSD,H1: Process already patched for the 2gb limit removal or we just can’t find the area to patch.
    2012.01.27 19:39:33 birt’s patch EURUSD,H1: You should check for a new patch at http://eareview.net/tick-data
    2012.01.27 19:39:33 birt’s patch EURUSD,H1: The patch you are running was not tested with this build so it may or may not work.
    2012.01.27 19:39:33 birt’s patch EURUSD,H1: MT4 build 409 detected.
    2012.01.27 19:39:33 birt’s patch EURUSD,H1 inputs: NoFXTOverwrite=false; WARNING1=”The 2GB limit removal only works in Windows 7 and Vista.”; Remove2GBLimit=true; WARNING2=”Using variable spread with a non-variable spread FXT does NOT work.”; RealSpread=false;
    2012.01.27 19:39:18 birt’s patch EURUSD,H1: loaded successfully

    I already tried to load the patch prior to everything else, but still the same problem. And I still have the limitation of 2gb :-/

    Thanks for helping and best regards

  • #1900 written by Alex January 24, 2012 (3 days ago)

    I don’t know how but now it works and shows 99%. I have done all the same.
    Thanks for your help.
    When I test for m1 the horisontal bar under the modeling quality is red, is it ok ?

  • #1901 written by birt January 24, 2012 (3 days ago)

    Yes, that’s normal with tick data backtests.

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top