Downloading and parsing Dukascopy tick data with Birt’s PHP scripts

The Dukascopy data is available on the web in its raw form as files that span only 1 hour, so it becomes apparent that some tools are necessary to download and parse it. Before it was possible to get the data via any of the other methods, I made a series of scripts that I still use nowadays for downloading the free tick data available from Dukascopy. I’m a fan of the PHP simplicity, so I chose that to write the scripts. They’re not commercial quality code, but they work.

You can get the PHP script archive from the tick data downloads page.

You will find 4 scripts inside:

  • A script for downloading the Dukascopy data, suggestively named “download_dukascopy_data.php”. As a courtesy to Dukascopy who is graciously providing free data, the script does not attempt to download files that are already on your harddisk. However, it still requests missing files, so to avoid stressing their server please set the dates in the $currencies array at the beginning of the script to the date of your last download; they’re using the standard unix timestamps (epoch date, which is in essence the number of seconds since 01.01.1970). If you want to easily convert from a regular date to such a unix timestamp, you can use Epoch Converter, a very easy to use online tool.
  • A script for processing the downloaded data, which assumes that it is located in the same directory as the previous script and that the data was downloaded there (process_dukascopy_data.php); this one needs some parameters, run it without any for a description or check out the next script.
  • A small shell script that will process all the downloaded data available in .bat form for windows and .sh form for linux.

Windows download & convert to CSV how-to

Start by visiting the windows PHP download section and fetching the latest binary version as a zip file.
Once you’ve done that, unpack it to c:\php\ and also unpack the scripts from the script archive you downloaded in the same directory.
Rename c:\php\php.ini-development to c:\php\php.ini. If your folder does not contain a file named php.ini-development, use php.ini-dist or any other php.ini-something file you can find.
Edit c:\php\php.ini, search for
;extension=php_curl.dll
and remove the semicolon in front of the line and add an “ext/” in front of “php_curl.dll” so that it looks like this:
extension=ext/php_curl.dll
Save the file and exit.
If you run into a zip error and your PHP installation has an ext/php_zip.dll, also apply the method above for extension=ext/php_zip.dll.
Head to the 7-Zip download page and get the command line version. Unpack it and put 7za.exe in the same directory (c:\php\).
Click start->run and type
cmd
then click ok (or alternatively type cmd and hit enter in the windows 7/vista “search programs and files” box in the start menu).
Type
cd \php
in the command window.
Type
php download_dukascopy_data.php
Have a coffee. Have another coffee. Go sleep. Go to work. Go to the gym. Go to a club. Wait some more. I’m not kidding, it takes ages. You can check the progress by watching the currency pair directories get filled. If you get any strange errors, run the process again when it’s finished – it will only download any files that were missed in the first step due to network errors.

If you only want to download some of the currency pairs available, you can edit download_dukascopy_data.php and change the array at the beginning of the file. You can switch the currency pair download order or completely remove the pairs that you don’t want. The number next to each pair is the unix timestamp at which to start downloading; if you wish to start at a later point in time (the default is the earliest date available) you can use epochconverter.com to get the timestamp for your chosen date.

When the download is finished, assuming you wanted to get the EURUSD data up to 01.01.2012, you’d type
php process_dukascopy_data.php EURUSD 200702 201201 EURUSD.csv
and the output will be placed in EURUSD.csv.
Alternatively, you can type
process.bat
which will batch process all the currency data. It’s mostly safe to ignore the error spam at this step. Note: if you use process.bat or process.sh, you might have to update the ending dates in them to get the full data range!

This should be it, if everything went fine you should have your CSV files in the same c:\php folder and you should be ready to proceed with preparing your tick data for Metatrader 4.

Warning: make sure you have enough space on your harddisk. As of 2012, the downloaded tick files have over 20 GB and if you add up the size of the resulted CSV files you will be well past the 100 GB mark.

  • #102 written by mandroe September 6, 2015 (1 year ago)

    Hi Birt,
    When I want to convert bi5 file into csv using
    C:\php>php process_dukascopy_data.php EURUSD 201501 201508 EURUSD.csv
    the message say : Error:failed to extract [EURUSD/2015/00/01/01h_ticks.bi5
    why?can you help me?

    • #103 written by birt September 6, 2015 (1 year ago)

      Do you have 7za.exe in that folder?

      • #104 written by mandroe September 6, 2015 (1 year ago)

        Yes, in C:\php folder

  • #105 written by mandroe September 6, 2015 (1 year ago)

    Yes

    • #106 written by birt September 6, 2015 (1 year ago)

      I don’t know then, check that the file was downloaded correctly and try to manually extract it with 7za.exe.

      I suggest using something like the SQ Tick Data Downloader or Tickstory Lite instead, that might assist with your task.

      • #107 written by mandroe September 6, 2015 (1 year ago)

        I already use tick data downloader or tickstory which both are failed due to some memory issues. I’ll try to extract manually with 7za.exe.

        • #108 written by birt September 6, 2015 (1 year ago)

          In that case it sounds like there’s something very wrong with your computer.

          • #109 written by mandroe September 6, 2015 (1 year ago)

            Maybe. But I also try on another laptop with the same result. Extract manually with 7za also failed. Confuse 🙁

            • #110 written by birt September 6, 2015 (1 year ago)

              Both SQ Tick Data Downloader and TickStory Lite are working fine so you must be doing something wrong but I can’t be sure what it could be.

              • #111 written by mandroe September 7, 2015 (1 year ago)

                I think something wrong with the downloaded bi5 file with the script. All bi5 file have2 kb size and can not extracted with 7-zip. I try to compared it with 1 sample bi5 i found on internet that has 21 kb size and able to be extracted with 7-zip. I try to redownloaded a few times but give the same result.

                • #112 written by birt September 7, 2015 (1 year ago)

                  Take a look at the content of the file, that might indicate the problem, it could be an error HTML.

                  • #113 written by mandroe September 7, 2015 (1 year ago)

                    I think I know why. In my country many forex web are officially blocked including dukascopy. That’s why downloaded bi5 by script/TD/TS is corrupt thus can not be converted into csv. I try to use proxy to avoid the block and only TS succeed to download the right bi5 file and converted it. Script & TD still download the corrupt file. Many thanks for your help.

                  • #114 written by hello July 23, 2016 (4 months ago)


                    birt:

                    Take a look at the content of the file, that might indicate the problem, it could be an error HTML.

  • #115 written by franck May 9, 2016 (7 months ago)

    Hi,
    I am using the Tick Downloader from Strategy Quant to download M1, M15, …D1 CSV files and imported them into my MT4 to do Backtesting.
    Now I was expecting the HST files to grow, because I thought that the CSVs were imported into the Historie, which is not the case.
    Where is the data stored?
    In addition a Backtest on M15 now takes about 36 hours, whereas a Backtest for the same time period took about 1.5 h on my previous HST history files.
    What am I missing here and how can I convert a CSV into an HST file?

    Thanks,
    Franck

    • #116 written by birt May 9, 2016 (7 months ago)

      First of all, please note that my website deals with tick data backtesting, not with importing CSV files into HST. The CSV2FXT script that processes the CSV files into tick data FXT files will automatically generate the correct HST files. If you would like more info about that, I suggest getting a TDS trial and reading How to prepare your tick data for Metatrader 4

      Regarding the increased backtest duration, you probably had a lot less data in your HST files before the import. I’d also wager you’re backtesting an EA with virtualized code, that’s why it takes so long. It’ll probably take even longer with tick data. Just to get an idea if it’s your EA or something else, try backtesting the MACD EA on the same currency pair and timeframe and compare the duration.

      Finally, the data is stored in the MT4 data folder, in history\. To get there, head to File -> Open Data Folder.

      • #117 written by Franck May 9, 2016 (7 months ago)

        Thanks Birt, somehow I thought you provided the tick downloader tool to Strategy Quant, which is why I posted here in the first place… my mistake.
        Yes, I found the imported data in an almost 48 GB FXT file in the tester\history folder, which might explain the duration of the test.
        I will read into the basics and look at the TDS.
        Thanks,
        Franck

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <s> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top