Metatrader expert advisors dissected.
Tick data downloads
Note: if you wish to link to one of the downloads here, please link to the page and not directly to the file, the reason being that the files here get updated quite often, the older versions get archived & deleted and your link will end up leading to a 404 page.
Enabling tick data backtesting
Tick Data Suite v1.1.1 (updated 02.05.2012, compatible: MT4 build 225 – 419)
Birt’s patch script v0.21 (updated 21.02.2012, compatible: MT4 build 225 – 409)
Processing tick data
Birt’s CSV2FXT script v0.34 binaries (updated 29.04.2012)
Birt’s CSV2FXT script v0.34 source (you don’t need this unless you want to modify or recompile the DLL)
Downloading Dukascopy tick data
Birt’s Dukascopy tick data PHP scripts v0.25 (updated 22.03.2012)
Tim’s Dukascopier v0.4 (updated 26.03.2011)
Miscellaneous
Gain Capital data tools
Multiple MT4 instance loaders (only working up to MT4 build 409)
History file (HST) time shifter
Runtime links
.NET Framework 4.0 (required for Tim’s DukasCopier)
Visual C 2010 runtime (x86) (required for removing the 2GB limit on Windows XP/2003 Server with MT4 builds 405+)
Visual C 2010 runtime (x64) (required for removing the 2GB limit on Windows XP x64/2003 Server x64 with MT4 builds 405+)
- #3 written by birt March 4, 2012 (2 months ago)
Actually, you were right, it is the Dukascopier. The Dukascopy tick data storage format has changed and starting with the second half of Jan 2012, the older format is no longer available. Since currently the Dukascopier only “knows” about the older format, it will not download or process anything past that point.
- #4 written by Uzair March 4, 2012 (2 months ago)
- #5 written by birt March 4, 2012 (2 months ago)
Like I mentioned, I am not the one making the Dukascopier, Tim is. If he supplies an update, sure, I will upload it. In the meantime, you can contact him about an update (his email is in the about tab of the app) – my PHP scripts are updated to work with the new data format so he could just port my code. However, seeing that there was no update to it in a rather long time, I’m not sure if the project is still being maintained.
- #6 written by Fred February 16, 2012 (3 months ago)
Replacing the download URL from …*.bin to …*.bi5 redirects to a bin file. These work past end of Jan, however these bin files are totally different. They are compressed using 7zip. Having uncompressed this file, I am unable to reverse engineer the format. Matches 100% mpeg, but I can’t quite believe that…
- #7 written by birt February 26, 2012 (2 months ago)
The format is not a problem, but PHP doesn’t have anything readily available to decompress LZMA (7zip) so the PHP scripts are going to have to wait.
The format (all integers are big endian):
4 bytes – delta time (from start of the hour) in milliseconds, stored as a 32 bit int
4 bytes – ask price multiplied by 10 * digits (divide by that to get the value), stored as a 32 bit int
4 bytes – bid price multiplied by 10 * digits, stored as a 32 bit int
4 bytes – ask volume, expressed in 100k units (so having 1 here means 100k), stored as a 32 bit float
4 bytes – bid volume, same as ask volume, stored as a 32 bit float- #8 written by Dop March 1, 2012 (2 months ago)
- #9 written by birt March 1, 2012 (2 months ago)
Well, I know it’s LZMA because I was able to unpack them using LZMA tools. As for the data format, it was quite easy to guess and it is specified in the comment above yours.
I just updated the PHP scripts to v0.21 which are able to seamlessly work with the new format, however the processing script requires the command line 7-Zip executable when running on windows.
- #10 written by nicco February 26, 2012 (2 months ago)
- #11 written by alex February 26, 2012 (2 months ago)
I have the same problem:
Hi Everyone,
I use the jForex dukas app to download bin files and create csv
with the new script I get the error:
Unable to identify the ask and bid colums
“Bad CSV format”I have updated all scripts and made sure the dukas copy app is current – still get the error
is there some other patch we need?
- #13 written by nicco February 27, 2012 (2 months ago)
- #15 written by nicco February 27, 2012 (2 months ago)
I see the problem. jForex Dukas is not using commas in their CSV production:
Image: http://www.fastmage.com/dukas.PNG- #16 written by birt February 27, 2012 (2 months ago)
Please (re-)read the Downloading Dukascopy tick data with JForex guide, especially the part about the delimiter.
- #17 written by nicco February 27, 2012 (2 months ago)
- #19 written by MachineGhost March 2, 2012 (2 months ago)
There seems to be a problem with the process dukascopy PHP script and data holes with the new format:
Warning: 0 sized AUDNZD/2009/00/07/00h_ticks.bi5
Warning: 0 sized AUDNZD/2009/00/07/01h_ticks.bi5
Warning: 0 sized AUDNZD/2009/00/07/02h_ticks.bi5
Warning: 0 sized AUDNZD/2009/00/07/03h_ticks.bi5
Warning: 0 sized AUDNZD/2009/00/07/04h_ticks.bi5
Warning: 0 sized AUDNZD/2009/00/07/05h_ticks.bi5
Warning: 0 sized AUDNZD/2009/00/07/06h_ticks.bi5
Warning: 0 sized AUDNZD/2009/00/07/07h_ticks.bi5
Warning: 0 sized AUDNZD/2009/00/07/08h_ticks.bi5
Warning: 0 sized AUDNZD/2009/00/07/09h_ticks.bi5
AUDNZD/2009/00/07/10h_ticks.bi5
Error: failed to extract [AUDNZD/2009/00/07/10h_ticks.bi5] - #26 written by Eugene March 2, 2012 (2 months ago)
- #28 written by Jack Lee March 4, 2012 (2 months ago)
- #29 written by birt March 4, 2012 (2 months ago)
The loaders are not even supposed to prevent rebuilding the FXT file for over 1 year now. That functionality is now in the Tick Data Suite or alternatively in the Birt’s patch script. Same goes for the 2GB limit removal.
- #30 written by Nicolas March 4, 2012 (2 months ago)
- #31 written by birt March 4, 2012 (2 months ago)
An FXT using Dukascopy data is some 2-3 GB per currency pair per timeframe. The CSV files are 5-7 GB per currency pair. The bin files for all currencies are over 10 GB, not sure exactly how big because it takes ages to measure their size given the number of files; likely they’re around 15 GB.
If you intend to use SSD, I would advise downloading the bin files with the PHP scripts and simply creating the CSV + FXT for the currency pair and timeframe required whenever you need them and deleting them when you’re done. This way you could probably manage it without problems with a 64 GB SSD drive.
- #32 written by jerome March 9, 2012 (2 months ago)
- #34 written by jiva March 21, 2012 (1 month ago)
- #39 written by niks March 22, 2012 (1 month ago)
- #40 written by birt March 22, 2012 (1 month ago)
Please read the feature matrix. In short, the script won’t work on builds past 409. If you need to run build 416, you need the Tick Data Suite which completely replaces the script. A trial version is available.
- #41 written by Pya March 27, 2012 (1 month ago)
- #42 written by birt March 27, 2012 (1 month ago)
The Tick Data Suite has nothing to do with the minimum lot size.
The minimum lot size in a tick data backtest is the minimum lot size that is found on the brokerage that you used to create your FXT with CSV2FXT.
Simply put: if you use a broker that has a min lot of 0.1 when running CSV2FXT, then your tick data FXT will also have a min lot of 0.1; if you use a broker with a min lot of 0.01, this will be reflected in the FXT and you will have a min lot of 0.01 in your tick data backtests.
It is also possible to modify the minimum lot in an existing FXT file but it involves a hex editor and looking up the offset in FXTHeader.mqh, take a look at the FAQ for more info. I don’t recommend attempting this procedure if you’re not familiar with hex editing.
- #43 written by Pya March 27, 2012 (1 month ago)
- #45 written by STan Fralick April 9, 2012 (1 month ago)
Hi Bert, thanks for the scripts and the very useful web site.
I have been using Dukascopier to download Dukas data, but as you stated it no longer works, so I downloaded the latest version of download_dukascopy_data.php(v0.25). Also downloaded and installed the latest version of php (5.4.0). Set up the ini file, and tried to run the download. I get errors indicating that several entry points in the php5.dll (and other dll’s) are not there. When I check the dll’s, sure enough there are no such entry points. I tried ver 5.3.10 and ver 5.2.10 of php and got similar errors (not exactly the same, but entry points are missing). I have php running under Vertrigo to support an Apache server, but I have turned Vertrigo off, and that version of php is in a different directory, so I don’t think I have interference. Have I missed some critical explanation?( My OS is XP).
- #47 written by STan Fralick April 10, 2012 (1 month ago)
Using the ver 5.4.0 php I get the following error twice:
“The procedural entry point -mysqlnd__end_psession could not be located in the dynamic link library php5.dll”
Then I get the error:
“The procedural entry point php_mb_check_encoding_list could not be located in the dynamic link library php mbstring.dll”
Then the script chmods the folders for the first data, and exits.Sorry I cannot paste the error notices, because I cannot capture them on the clipboard. I have images but no way to upload. I checked the wording carefully.
When I use earlier versions of php, I get similar errors, but the specific entry points are changed.
- #48 written by birt April 10, 2012 (1 month ago)
That means the DLLs are not in the path. I would suggest following the guide exactly as it is written, using C:\php as the installation path and executing php while your current folder is c:\php.
Alternatively, you could change your PATH variable, see http://windows.fyicenter.com/view.php?ID=51 and http://windows.fyicenter.com/view.php?ID=60 for more information.
- #49 written by STan Fralick April 10, 2012 (1 month ago)
Thank you for your help.
I have done exactly as you suggest. php is installed at c:\php. When I run cmd, I cd to the c:\php directory and run the script from there. I have modified the php.ini file so that the reference to the curl dll is now ‘extension=ext/php_curl.dll’. I have copied the php5.dll to both the php directory and the php\ext directory. I get the same resultant errors. When I examine the entry points into the dll’s using DLL viewer, I find that there are no such entry points. e.g. here are the listed entry points alphabetically near the missing entry:
mysqlnd_connect 0x10147e80 0x00147e80 715 (0x2cb) php5.dll C:\php\php5.dll Exported Function
mysqlnd_cset_escape_quotes 0×10148890 0×00148890 716 (0x2cc) php5.dll C:\php\php5.dll Exported Function
mysqlnd_cset_escape_slashes 0×10148930 0×00148930 717 (0x2cd) php5.dll C:\php\php5.dll Exported Function
mysqlnd_debug_init 0x10149cb0 0x00149cb0 718 (0x2ce) php5.dll C:\php\php5.dll Exported Function
mysqlnd_debug_std_no_trace_funcs 0x1053e9c0 0x0053e9c0 719 (0x2cf) php5.dll C:\php\php5.dll Exported Function
mysqlnd_empty_string 0x103d6bfc 0x003d6bfc 720 (0x2d0) php5.dll C:\php\php5.dll Exported Function
mysqlnd_field_type_name 0x101471e0 0x001471e0 721 (0x2d1) php5.dll C:\php\php5.dll Exported Function
mysqlnd_fill_stats_hash 0×10155760 0×00155760 722 (0x2d2) php5.dll C:\php\php5.dll Exported Function
mysqlnd_find_charset_name 0×10148850 0×00148850 723 (0x2d3) php5.dll C:\php\php5.dll Exported Function
mysqlnd_find_charset_nr 0×10148820 0×00148820 724 (0x2d4) php5.dll C:\php\php5.dll Exported FunctionSince these scripts work for so many users, I am quite sure that I am doing smething foolish, and will have a red face when I discover just what it is, but for now I am baffled.
- #50 written by birt April 10, 2012 (1 month ago)
Well, your problem is not with the scripts but with PHP itself. I’ve never had such problems before so I’m guessing it’s something specific to your machine, possibly having something to do with your other PHP installation. I would suggest posting your issue on a PHP forum such as http://www.phphelp.com/forum/. I suggest checking your environment vars for any potential issues.
In the meantime, you could download the data with JForex or via the Dukascopy website.
- #52 written by STan Fralick April 10, 2012 (1 month ago)
- #53 written by iDouble April 11, 2012 (1 month ago)
Build 409, anyone?
All of a sudden, my build 409 now cares about the fact that I renamed the liveupdate file, when it never cared about it before – bizarre. Now, 409 starts and then immediately shuts down. If I rename liveupdate.old back to liveupdate, 409 launches, at then attempts to upgrade itself (of course).
My stupid broker gave me a link, promising me that it was their previous 409 download – when in reality, it was their 419. After installing that, I can’t seem to find a 409. Everything that says its 409, install as 419. What pain in the neck.
Does anybody know where you can get 409, and have it actually install as 409 and not 419?
Much appreciated! Very frustrated today.
Thanks,
iD - #54 written by iDouble April 11, 2012 (1 month ago)
Well, turns out that all you need to do is locate a copy of an old Terminal.exe for 409, and replace the current Terminal.exe file already on your file system. Rename the current Terminal.exe, to something like Terminal.old, and then simply drop the 409 version of the same file into its place. You might also want to rename liveupdate, to something like liveupdate.old, so MT4 won’t attempt any updates.
Hope that helps someone.
- Comment Feed for this Post
Didn't find any related posts :(
Dukascopier is broken — won’t download any data past the end of Jan 2012.