Metatrader expert advisors dissected.
Tick data changelog
- 02.05.2011 – Tick Data Suite v1.1.1 released: extends the 4GB FXT size limitation to over 200 GB; it is now also possible to run the Tick Data Suite with a non-administrator account; several minor isolated bugs that have been corrected.
- 29.04.2011 – Tick Data Suite v1.1.0 released: added automatic update checking; added configuration helper application that helps copy the TDS executable to multiple MT4 folders, allows easily changing the backtest start delay factor and also permits interacting with the registration; all files are now code signed to prevent further false positives with some antivirus programs.
- 29.04.2011 – CSV2FXT v0.34: fixed major bug related to parsing the histdata.com date format; added Australian DST (AEDT); replaced most alerts with message boxes; added question for automatic moving of the HST & FXT files to the proper locations; added overwrite questions for readonly FXT files in experts/files and in tester/history.
- 15.04.2011 – CSV2FXT v0.34 (not yet released): added code that should prevent negative spreads when using real spread (even in conjunction with a negative spread padding).
- 09.04.2011 – CSV2FXT v0.34 (not yet released): Russian DST added; compiler warning about long line fixed; when using fixed spread, duplicated ticks within the same minute are now filtered.
- 06.04.2011 – Tick Data Suite v1.0.7 is out. It fixes a minor issue with MT4 build 419 (unable to start multiple instances from the same folder).
- 28.03.2011 – CSV2FXT v0.33 is now up. It fixes a bug in v0.31 and v0.32 where a floating point value equality check was not working correctly, resulting in some error messages coming up incorrectly and the resulting files being potentially broken.
- 27.03.2011 – CSV2FXT v0.32 only adds a few more details (the ask & bid quotes) to the errors printed in the experts log when tick prices are incorrect (the checks added in v0.31).
- 24.03.2011 – Tick Data Suite v1.0.6 is out. It adds compatibility with Metatrader 4 build 418.
- 23.03.2011 – CSV2FXT v0.31: added safeguard for the tick prices – if the ask price is different from the bid by more than 10% there must be an error in the CSV; similarly, if the next tick differs with more than 10% from the previous tick, there must be an error in the source data. This replaces the check from v0.30 which did not work properly in all situations.
- 22.03.2011 – The Dukascopy PHP scripts are now at v0.25. Changes: a small correction that enables correct operation with the new tick data format on MacOS.
- 19.03.2011 – CSV2FXT v0.30 uploaded. It adds error checking for incorrect prices in the CSV file. The Dukascopy history page seems to have some issues with this sometimes and such problems will now be reported in the log and faulty ticks will be skipped. To be more specific, for example I’ve seen an EURUSD tick where the ask price was around 1.3 and the bid was 88.something.
- 07.03.2011 – Tick Data Suite v1.05 released. Adds compatibility with MT4 build 416 and fixes an issue with file locking that would prevent reading the log files while MT4 was running.
- 07.03.2011 – CSV2FXT v0.29 released. Fixes a problem where Bid would be reversed with Ask in some rare situations where the first tick in the file had identical Bid and Ask prices (for instance USDJPY exported by JForex).
- 04.03.2011 – The Dukascopy PHP scripts are now at v0.24. Fixed two more bugs, one of which was a debug setting forgotten in it. Three versions in a single day should teach me to avoid releasing stuff in a hurry (which is what happened with v0.21).
- 04.03.2011 – The Dukascopy PHP scripts v0.23 are now up. This version fixes another couple of bugs involving handling the bi5 files on Windows, thanks to Alex for pointing out one of them.
- 04.03.2011 – The Dukascopy PHP scripts v0.22 have been uploaded. This version fixes a bug handling the bi5 files on Windows, thanks to Eugene for pointing it out.
- 01.03.2011 – Version 0.28 of CSV2FXT is out. It fixes a minor bug that prevented setting the generated FXT files readonly when Metatrader 4 was started by its shortcut.
- 01.03.2011 – Version 0.21 of the Dukascopy PHP scripts has been released. It adds full compatibility with the new Dukascopy tick data format. It requires the 7-Zip command line executable on Windows and lzma or xd on Linux.
- 27.02.2011 – Tick Data Suite v1.0.4 released. Fixes bug related to saving backtest reports in Metatrader 4 build 414.
- 26.02.2011 – Tick Data Suite v1.0.3 released. In spite of a whole week of beta testing, the MT4 build 414 changes also introduced a bug that made the TDS not work with builds 402 and below, which was addressed in this hotfix. In addition to that, the Tick Data Suite will let the MT4 terminal start minimized if requested by a 3rd party application (WFA does that) and it should also work on computers for which it previously said “Unable to retrieve hardware ID”.
- 25.02.2011 – Updated CSV2FXT to v0.27. This version knows how to handle symbols that are not currency pairs (CFDs, indices etc.).
- 25.02.2011 – Tick Data Suite v1.0.2 released. Adds support for MT4 build 414.
- 22.02.2011 – Many people link directly to the files instead of the downloads page and this leads to other people using terribly outdated versions of things, so I removed everything that was outdated in the downloads folder. All such links will now result in a 404 error and this will continue happening in the future with all files that are linked directly.
- 21.02.2011 – Updated CSV2FXT to v0.26. Now it knows how to read date formats with no separators (histdata.com).
- 21.02.2011 – Updated the Birt’s Patch script to v0.21. The only change is that now it correctly reports not working with MT4 builds newer than 409 and aborts.
- 21.02.2011 – Tick Data Suite v1.0.1 beta uploaded. Works with MT4 build 414.
- 15.02.2011 – Tick Data Suite v1.0.0 was released.
- 14.02.2011 – Updated CSV2FXT to 0.25. Fixed minor bug that would allow the script to run even if no FXT could be opened.
- 10.02.2011 – Updated CSV2FXT to 0.24. Added options to create FXT files for multiple timeframes at once.
- 10.02.2011 – Added HistoryShift, a small script that generates new HST files with a shifted date.
- 10.02.2011 – Updated CSV2FXT to 0.23. Added the TimeShift option (which translates the tick data time 28 years in the past).
- 07.02.2011 – Updated CSV2FXT to 0.22. Fixed issue that prevented the progress indicator display when MT4 was started via its shortcut.
- 07.02.2011 – Updated the CSV2FXT script to 0.21. It now works around a bug in the MT4 installer that fails to set the “Start in” directory of the MT4 shortcuts; starting MT4 by one of its shortcuts would previously result in “function ‘CsvIsLineEnding’ call from dll ‘CsvReader.dll’ critical error”. The CsvReader source was also updated with a small fix that ensures the buffer is filled when seeking (needed for files with a header row).
- 04.02.2011 – The PHP scripts for downloading and parsing Dukascopy data are now v0.20 as well. The candle data script was removed from the archive since it was mostly confusing people and it might get readded at a later time as a separate download. Other than that, the only change was the
strftime("%D %T")which resulted in a warning on Windows. Oh, almost forgot, the bat & sh scripts now process up to 2013 by default. - 04.02.2011 – Birt’s patch is now v0.20. The only change is that it now complains if DLL calls are not enabled.
- 04.02.2011 – Added separate archive with the CSV2FXT script source code and DLL source code.
- 04.02.2011 – Added CSV2FXT script (v0.20) that is able to figure out the CSV format regardless of the tick data source. The script uses a DLL to read the CSV files and thus bypasses the bug in MT4 build 409 that resulted in skipped years. Among other new features, a progress indicator is displayed on the chart and the output in the experts log is much more verbose.
- 04.02.2011 – Added versioning to all the scripts.
- 04.02.2011 – All the old FXT scripts (Dukascopy2FXT, Oanda2FXT, JForex2FXT) are obsolete as of now.
- 16.11.2011 – Fixed the crash in subsequent backtesting in builds 405 and above.
- 11.09.2011 – The changes in builds 405 & 406 were major from the point of view of the tick data tools. As a consequence, a new generic loader was needed and this one works only with builds 405 and up; the old one is still in the “old” folder inside the archive in case anyone needs to use it on builds 402 and below. The MQL scripts were also updated to work with the new builds and as a side bonus, since I had to fully rethink the 2GB limit approach, I also made it run on Windows XP, but only for builds 405 and up and it requires the Visual C 2010 runtime; see above for the runtime download links.
- 04.06.2011 – Added generic loader because the whole thing with Metaquotes releasing 4+ different builds with the same build number was getting on my nerves. This should work for builds 228 and up, if you need a loader for an older build look in the old directory in the archive. First time you run the generic loader, it will find the offsets; don’t panic and follow the instructions. If you update your MT4 build, it will just find the offsets again. If it fails to find the offsets for a new build just post a comment here.
- 26.05.2011 – Added loader for yet another 401 build.
- 15.05.2011 – Added loader for build 402.
- 14.05.2011 – Added some more pairs to the PHP downloader script: USDSGD, AUDCAD, AUDCHF, CADCHF, EURNZD, GBPAUD, GBPCAD, GBPNZD, NZDCAD, NZDCHF, NZDJPY, XAGUSD and XAUUSD. The above-mentioned pairs are available starting from February 2010, with the exception of USDSGD (September 2008), XAGUSD (November 2010) and XAUUSD (May 2011). The batch scripts were also updated and the processing script was replaced with the workaround script because it seems that PHP bug is there to stay.
- 14.05.2011 – The patch script is now able to remove the 2GB FXT limit for builds 399 and above. So far it works for 399 and 401 and hopefully it’ll work on future builds, too.
- 14.05.2011 – Added loaders for build 399 and the two different 401 builds I ran into.
- 07.04.2011 – Metaquotes has released a new build 392 with the same size, that resulted in a CRC error in the loader. I updated the loader and disabled the CRC check. Both 392 builds should work with the same loader now, but you need to update it if you have the old one and get the CRC error.
- 26.03.2011 – Updated Dukascopier to v0.4, which contains some fixes. The issue with a comma as the decimal separator on some locales has been addressed.
- 17.03.2011 – Added loader for build 392.
- 14.03.2011 – Updated Dukascopier to v0.3.
- 11.03.2011 – Added yet another loader for build 388. Metaquotes persists in its nasty habit.
- 11.03.2011 – If the terminal is not connected, the FXT scripts will now wait for 15 seconds before exiting with an error message. This should allow them to be used with startup ini files.
- 02.03.2011 – Metaquotes has acquired a nasty habit as of late. They release a build (in this case 388) and several days later they probably encounter some bug that must be fixed, address it and proceed to release a new build with the same number instead of increasing it. Just like 229, there are now two different 388 builds. I’ve added a loader for the new one, marked 388a. I also added detection for this build in the patch – you don’t need to upgrade, though; the old one will work just fine.
- 01.03.2011 – Updated Tim’s DukasCopier to v0.2. This version adds candle data downloading.
- 28.02.2011 – Once again, the old patch worked with the new 388 build, saying “unknown version” in the log. Now it is also able to identify 388. If you don’t have a problem with the patch saying “unknown version”, you don’t need to update.
- 28.02.2011 – Added loader for build 388.
- 27.02.2011 – Added Tim’s DukasCopier.
- 24.02.2011 – Although the old patch works just fine with build 387, it would say “unknown version” in the log. That has been changed. There’s no need to update unless that message bothers you.
- 24.02.2011 – Replaced short PHP open tag with full tag in download_dukascopy_candle_data.php
- 24.02.2011 – Added loader for build 387.
- 16.02.2011 – The margin currency used in the FXT files is now the account currency. Please note that this might not work properly if you hide some pairs. As a rule of the thumb, make sure that all currencies involved in the backtest have a “link” to the base account currency. For example, if your account currency is AUD and you’re backtesting EURJPY, ensure that both EURAUD and AUDJPY are among your visible symbols.
- 15.02.2011 – In addition to setting the GMT offset of the resulting FXT, you can now set the DST according to the US or European legislation.
- 13.02.2011 – Fixed two bugs that were leading to an incorrect commission calculation.
- 11.02.2011 – The MQL scripts now perform a NormalizeDouble(price, Digits) on the data read from the CSV files, meaning that the FXT files generated and used on 4 digit brokers will no longer result in invalid price errors.
- 11.02.2011 – A scripts that allows processing Dukascopy candle data has been added. A script for downloading it was already in there.
- 09.02.2011 – The FXT scripts will print a message in the experts log when starting to process a new year of data. The readonly attribute is now set at the very end to allow printing & alerting even if DLL calls are disabled.
- 07.02.2011 – Added alternate PHP process script for people running into CLI crashes.
- 07.02.2011 – Added Leverage option to all FXT processing scripts.
- 19.09.2010 – Updated the FXT scripts to also write the first and last tick of the file as its start/end date, thus avoiding the 1970 dates displayed in the results.
- 21.04.2010 – Updated the process script to resume if the output file exists, effectively allowing incremental CSV generation.
- 16.11.2009 – Implemented the fix Futron posted in the comments section.
Hi Birt,
I have been perusing your webpage for more than a year. I just wanted to express my strong feeling of indebtedness and thankfulness for your extremely precious work and for the quality of the patches you provide us with. I am particularly impressed with your ability to keep up with the good work despite Metaquotes’ BAD BEHAVIOUR – their continuous modification of MT4 – version after version.
Just one question – Is there a place where you can download all the old version of MT4 other than Metaquotes (last time I checked some versions were missing).
MANY MANY THANKS!!!