Downloading Dukascopy tick data with JForex

Begin by registering a demo account with Dukascopy and start the JForex platform (you can of course register a live account, the data process is the same). Login using the data in the email you received (note that you don’t need the account ID to login) then go to the Tools menu and select Historical Data Manager. In the lower part of the window, the Historical Data Manager interface should appear; from now on, everything you have to do happens in that part of the window so you might want to enlarge it a bit. Proceed as follows:

  1. Select , (comma) in the Delimiter field. Don’t leave that field blank and don’t select the dot (.).
  2. For the Data Type field, select Ticks.
  3. In the Instrument pane, select all the symbols that you want to download the data for.
  4. Select the Date From and Date To of your choice. Note that the earliest date available for most major currency pairs is 2007.03.01.
  5. It’s safe to leave the Date Format field unchanged.
  6. If you want the data to be exported to a different location, click the Browse button.
  7. Hit Start and patiently wait until the progress indicator slowly (exactly how slowly depends on the amount of data you selected) crawls to 100%.
  8. Find the CSV files in the location of your choice.

Assuming everything went fine, you’re ready to proceed with preparing your tick data for Metatrader 4.

Note: JForex caches the data to your disk. If for some reason you intend to delete it, it can be found in C:\Users\[your username]AppData\Local\JForex\.cache on Windows 7. On XP/Vista, dig around in your user folder, it should be in a similar path but in Application Data instead of AppData.

  • #1 written by Jim February 14, 2012 (4 years ago)

    Many thanks Birt

  • #2 written by Robin February 23, 2012 (4 years ago)

    May I know what’s the filesize for one of these tick data CSV’s for one pair? It’s been 15 minutes since I started downloading and it’s still at 0%… Is the CSV like a few GBs big?

    • #3 written by birt February 23, 2012 (4 years ago)

      I wasn’t kidding when I said “crawls”. For example, the CSV for EURUSD for 2007-2012 is over 7 GB, but the size of the download should be much smaller because the files are compressed.

  • #4 written by LogicaLucidity March 28, 2012 (4 years ago)

    I have been using this method for several months and have not had any issues thus far. All the credit and thanks goes to Brit. I recently pulled GBPUSD data from Jan 09 – Jan 12, data that I have pulled in the past. This was the first time I used your new CSV2FXT script. I received an alert stating ‘Possible error: Large gap after 2009.06.12 20:59:53 (6.0 days). I have never received one of these before and assume they are new to the script. After looking back at previous back tests using this same time period I noticed that those 6 days were there before. Assuming a mistake, I pulled the data again and used the CSV2FXT with the same result. I then moved to a new PC and made sure to clear the cache located in C:\Users\[your username]AppData\Local\JForex\.cache and cleaned up java. I continue to be missing those 6 days no matter the way I approach it even though I had them before. I then decided to try the Dukascopy historical data page and find that no matter the PC I am on the downloading bar freezes at 8%. If anyone has any idea how this is possible please speak up. Thank you.
    LL

  • #5 written by L. April 9, 2012 (4 years ago)

    Hi!

    I’ve tried to dowload tick data according your instructions but the “start” button is grey and I can’t click on it. Can someone tell me what could be wrong?

    • #6 written by birt April 9, 2012 (4 years ago)

      Make sure that you configured everything (start & end date, currency etc.).

      • #7 written by FIBO19 May 29, 2012 (4 years ago)

        I have exactly the same problem. I`m sure that I configured everything according your instructions, but the “Start” button stays grey.

        • #8 written by FIBO19 May 29, 2012 (4 years ago)

          Now i found the reason. I forgot to enlarge the Historical Data manager-window.

  • #9 written by LogicaLucidity April 9, 2012 (4 years ago)

    To elaborate on my post above… When running the CVS2FXT Script on EURUSD Data from April 1 2007 to April 1 2012 I get the exact same gap, โ€˜Possible error: Large gap after 2009.06.12 20:59:53 (6.0 days)’. After talking to several people they too are confirming this gap. It has been suggested that the data from JForex was altered after they changed the tick format (was .bin before, now it’s .bi5). I do not know if that was the actual cause. I have yet to determine how many Pairs share this exact 6 day gap. At this point I assume all pairs.

    • #10 written by birt April 9, 2012 (4 years ago)

      I’ve just replied to one of the TDS customers who experienced the same problem. It looks like that week was “lost in translation” when they converted the files from .bin to .bi5. I still have old bin files on my Linux box up to January 2012 and I was able to verify that I have the data for that particular week but that doesn’t really do much good.

      I would suggest verifying whether that particular week is missing from any other currency pair and emailing Dukascopy about it. If they truly lost those bin files (or any bin files at all for that matter), they could contact me and I could provide the missing data for any of their currency pairs.

      • #11 written by LogicaLucidity April 9, 2012 (4 years ago)

        I will pull data from April 1 2007 to April 1 2012 forming a fresh Cache. I will run CVS2FXT on each one confirming whether or not this gap is universal. I will take my findings to Dukascopy and await a reply from them. Thank you Birt.

        • #12 written by birt April 9, 2012 (4 years ago)

          You don’t need to run CSV2FXT or even export to CSV files to check if the gap is there. You can simply browse your JForex cache folder and check if the files exist or not. Pay attention to the file structure – if it matches the structure on the website, January is month 0 not 1.

  • #13 written by LogicaLucidity April 9, 2012 (4 years ago)

    If I were only looking for that single common hole you would have saved me a lot of work. Running the CSV2FXT script will enable me to verify that there are not any other large holes that certain pairs share that have appeared in the data since the format change. I want to have a complete picture before I contact them to avoid going to them twice. It is more work, but if I find a second common hole among other pairs I will be glad I did it.

    • #14 written by birt April 9, 2012 (4 years ago)

      Someone just told me via email that there is a hole at the same point in the AUDUSD data.

    • #15 written by birt April 9, 2012 (4 years ago)

      If you contact them, you can also let them know that they have some big problems on 01.04.2007 (iirc) in their USDJPY and EURJPY data.

  • #16 written by LogicaLucidity April 22, 2012 (4 years ago)

    Hello Birt,
    You will have to be more specific about the ‘big problems on 01.04.2007 (iirc) in their USDJPY and EURJPY data.’
    I did not find any.

    I am hoping that you might confirm that this 6 day hole I am getting with every pair was not there before the format change.

    Is there anyway you can do that? You are the only one I have talked to who has a cache prior the format change.

    You are free to email me.

    I pulled all data for all 22 pairs for as far back as possible for each up to April 1, 2012. (90+GB)

    I ran your CVS2FXT script on each one.

    Every pair that has data available during June of 2009 has the exact same 6 day gap.
    Other pairs suffer from Gaps but none that are in common.

    This 6 day gap was not there before the format change on several of the pairs.
    I can only assume that it was not there on the rest as well.

    I will contact Dukascopy and inform them of the error and hope for a response.

    Here are the results:

    AUD/CAD Date Span:(2010.02.16-2012.04.01)
    No Gaps

    AUD/JPY Date Span:(2007.03.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    AUD/NZD Date Span:(2008.12.22-2012.04.01)
    large gap after 2008.12.22 16:25:03 (15.0 days).
    large gap after 2009.06.12 20:58:35 (6.0 days).

    AUD/USD Date Span:(2007.03.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    CAD/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:51 (6.0 days).

    CHF/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    EUR/AUD Date Span:(2007.3.30-2012.04.01)
    large gap after 2007.06.01 20:59:36 (24.0 days).
    large gap after 2007.06.26 08:03:43 (19.0 days).
    large gap after 2009.06.12 20:59:38 (6.0 days).

    EUR/CAD Date Span:(2008.09.23-2012.04.01)
    large gap after 2009.06.12 20:59:38 (6.0 days).

    EUR/CHF Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:47 (6.0 days).

    EUR/GBP Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:53 (6.0 days).

    EUR/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    EUR/USD Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    GBP/CHF Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:39 (6.0 days).

    GBP/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:53 (6.0 days).

    GBP/USD Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:53 (6.0 days).

    NZD/USD Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    USD/CAD Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:52 (6.0 days).

    USD/CHF Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:47 (6.0 days).

    USD/HKD Date Span:(2010.10.15-2012.04.01)
    large gap after 2010.11.10 16:33:01 (158.0 days).

    USD/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:51 (6.0 days).

    USD/MXN Date Span:(2010.10.15-2012.04.01)
    Pair not availible on my HoTForex Demo MT4 (409) platform.

    USD/SGD Date Span:(2008.09.28-2012.04.01)
    large gap after 2008.09.29 07:10:36 (6.0 days).
    large gap after 2008.10.06 16:35:24 (13.0 days).
    large gap after 2008.11.03 15:37:51 (647.0 days).

    LL

    • #17 written by birt April 22, 2012 (4 years ago)

      Those files have a size of 0 on the Dukascopy site, both the new format (bi5) as well as the old one (bin). In my download cache, they are fine. I will contact Dukascopy and offer to send them the files if they need them.

      • #18 written by LogicaLucidity April 22, 2012 (4 years ago)

        I have contacted them as well detailing out all I know.
        Hopefully they will reply.
        Thank you Birt.
        LL

        • #19 written by LogicaLucidity April 25, 2012 (4 years ago)

          Dukascopy’s Response to my concerns:

          Greetings,

          We do not provide any services for data download in a BIN or BI5 format.

          Best regards,
          Dukascopy Support

          I replied asking them to reconsider… explaining it in even greater detail.

          There is only so much one can do.

          • #20 written by birt April 26, 2012 (4 years ago)

            Well, at least you got a reply… I didn’t get one yet.

            Did you tell them that you downloaded the data using their JForex client? Also, did you tell them that downloading the data for that period via the website doesn’t work either?

            • #21 written by LogicaLucidity April 26, 2012 (4 years ago)

              Yes and yes. I went into elaborate detail.

              I provided links to their own site where the JForex client is presented.

              I really do not know what steps to take now.

              We have the facts. We just need someone to listen.

  • #22 written by Future April 30, 2012 (4 years ago)

    i’v got the same problem on EUR/USD H1

    2012.04.30 17:59:01 CSV2FXT EURUSD,H1: Alert: Possible error: large gap after 2009.06.12 20:59:48 (6.0 days).

    • #23 written by birt April 30, 2012 (4 years ago)

      As stated above, those files are broken (0-sized) on the Dukascopy website.

  • #24 written by LogicaLucidity May 4, 2012 (4 years ago)

    BOOMSHAKALAKA!!!
    My further attempts have finally paid off!

    Dukascopy’s most recent response:

    Greetings,

    Thak you for the reporting the problem!

    There are no any connections with a switching from BIN to BI5 format.
    In any case, the data will be restored as soon as possible.

    Best regards,
    Dukascopy Support

    • #25 written by LogicaLucidity May 8, 2012 (4 years ago)

      Here is their reply about when things will be fixed and if Gold and Silver will be restored…

      Greetings,

      We are going to repair all data we provide.
      It has to be done next week.

      Best regards,
      Dukascopy Support

      I Hope this is the end of this.

      Thank you Birt for all you have done,
      LL

  • #26 written by Zerit0 May 23, 2012 (4 years ago)

    Hi Birt, Hi LogicaLucidity

    Just got the same error on pair EURUSD “large gap after 2009.06.12 20:59:48 (6.0days) on D1.

    So I guess the “week” for dukascopy is still on course in order to fix the gap lol… have you tried the another pairs with the same issue or is there any news from Dukascopy?

    Best regards, and Birt thanks a lot for all your support, ๐Ÿ˜‰

    • #27 written by LogicaLucidity May 23, 2012 (4 years ago)

      On Monday I also ran EUR/USD through the CSV2FXT script after pulling it into a fresh cache.

      I too got the Possible error: large gap after 2009.06.12 20:59:48 (6.0 days).

      It looks like that none of these 6 day gaps were addressed.

      I only assume that they ‘tried’ to fix the issue given that the number of pairs has increased from 22 to 52 pairs.

      I gave them 2 weekends after telling me that they would fix it ‘next week’.

      I sent a reply to the on going conversation they have been having with me 2 days ago addressing this .

      I have not received a reply. I will try again at the end of the week.

      LL

  • #28 written by luca May 28, 2012 (4 years ago)

    hi guys all

    i can still confirm on audnzd cross , the gap mentioned is still present

  • #29 written by LogicaLucidity June 2, 2012 (4 years ago)

    New reply from Dukascopy:

    Greetings,

    The data will be restored as soon as it will be prepared and checked on our side.
    We cannot predict precisely the restore time. From the current point of view, it would be done next week.
    We are waiting for a response from the DWH side now.

    Best regards,
    Dukascopy Support

  • #30 written by Chris June 9, 2012 (4 years ago)

    Hi Birt

    Do you know if there is a problem with JForex as I can’t seem to get the “Start” button to be enabled on Historic Data Manager. I have followed all your instructions, but it still remains greyed out and even tried it on a another computer with same outcome. All I can think of they don’t allow this for demo users anymore.
    As someone else mentioned it’s pointless using the website data feeds as they always freezes – mine go to about 70% and remained there even 12 hours later – have reported by these problems. However, I was hopeful you might know what I have done wrong or could test it or someone else to see if you have the same result
    Btw thank you for your awesome website and user guides/software it is highly appreciated and can’t wait to make the most out of my trading experience by using what is on to my advantage once of course can obtain tick data. ๐Ÿ™‚

    Thank you,

    Chris.

  • #31 written by Chris June 9, 2012 (4 years ago)

    @ FIBO19 – you sir are a legend. Thank you. Please ignore my last message as have it working now (I had also forgot to enlarge the window) in order to select the instrument – instead was selecting instrument from the window to the left hand side.

    @ Birt – keep up the outstanding work! ๐Ÿ™‚

  • #32 written by Chris June 9, 2012 (4 years ago)

    I give-up thought it was working.. now “Start” is available, but clicking on it hasn’t done anything.. CPU in task manager isn’t even moving so clicked again and remains same. Anyone else having problem? =/

  • #33 written by Chris June 9, 2012 (4 years ago)

    Sorry to spam your site – feel free to delete these. It’s all working now for some reason about closing/restarting it then came up with the terms of conditions (agree/disagree) message before obtaining the information. Please keep that in mind should any of you also have the problem.

  • #34 written by FIBO19 June 9, 2012 (4 years ago)

    Hi Chris,
    by pushing START the disclaimer appears and in the blue bar is written: Downloading startet.
    But after 2 hours it still stands on 0%.
    I closed and opened the platform several times but the terms of conditions massage (agree/disagree) didn`t came up.
    So I still have problems with downloading Duka-Tickdata from Jan. 1. 12 till Jun. 9.12.
    Can you help please ?

  • #35 written by Chris June 9, 2012 (4 years ago)

    Hi Fibo.

    The disclaimer will come up once you have clicked on “START” after of course selecting the instrument and setting up the other parameters. As you mentioned previously maximize the “Instruments” window so it fits the whole box as it’s easier to see everything on page. Should the disclaimer not appear try restarting the application after a couple of minutes – logout correct and exit application before re-opening a second time. I would assume like me you click on the JForex link to load the application from on the website, that way you know it’s the latest version. Finally, the download won’t officially start until you click on “Accept” on the disclaimer. Try this and let me know what happens?

  • #36 written by Chris June 9, 2012 (4 years ago)

    FX Marketplace Platform
    Version 2.17.7
    [*] JForex

    There isn’t too much can say to help except the version number and that “disclaimer” no longer appears after accepting it once. Should everything else my advice would be to try make a second “Demo” account to force disclaimer up and take it from there.

    Let me know how you get on

  • #37 written by LogicaLucidity June 26, 2012 (4 years ago)

    New reply from Dukascopy.

    Greetings,
    Unfortunately we cannot provide you with a precise date when the data will be restored.
    Best regards,
    Dukascopy Support

    This was in response to an email asking if the issue would ‘ever’ be fixed.
    At this point I feel that the data will not be fixed.

    It has been 3 months…….Dukascopy has Failed.

    LL

    • #38 written by rajiv July 14, 2012 (4 years ago)

      Its available here
      from 3-2007 to 12-2011
      in CVS zipped form 6.56 GB in 600 MB

      http://

      Edited by Birt: none of the free tick data providers (including Dukascopy) allows data redistribution. If you’re willing to breach the agreements that’s your problem, but please do not post links to such content here.

  • #39 written by LogicaLucidity September 21, 2012 (4 years ago)

    As you may have read up above, AUD/USD use to look like this…

    AUD/USD Date Span:(2007.03.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    Things have changed a bit…

    I have only ran two pairs and they both look similar…. Swiss Cheese.

    For example, this is what AUD/USD looks like now…

    AUD/USD Date Span: (2007.04.01-2012.09.16)
    Possible error: gap after 2012.06.12 21:53:50 (5.0 hours).
    Possible error: gap after 2010.06.17 23:20:48 (6.0 hours).
    Possible error: gap after 2009.12.31 21:59:55 (72.0 hours).
    Possible error: gap after 2009.12.24 21:59:50 (72.0 hours).
    Possible error: gap after 2009.06.25 16:49:49 (15.0 hours).
    Possible error: gap after 2009.06.12 20:59:48 (161.0 hours).
    Possible error: gap after 2009.05.11 03:14:07 (3.0 hours).
    Possible error: gap after 2008.12.31 19:59:42 (26.0 hours).
    Possible error: gap after 2008.12.31 19:59:42 (26.0 hours).
    Possible error: gap after 2008.12.24 22:00:00 (24.0 hours).
    Possible error: gap after 2008.12.24 22:00:00 (24.0 hours).
    Possible error: gap after 2008.08.08 07:07:42 (4.0 hours).
    Possible error: gap after 2008.08.08 07:07:42 (4.0 hours).
    Possible error: gap after 2007.12.31 17:00:03 (29.0 hours).
    Possible error: gap after 2007.12.31 17:00:03 (29.0 hours).
    Possible error: gap after 2007.12.24 17:00:29 (36.0 hours).
    Possible error: gap after 2007.12.24 17:00:29 (36.0 hours).

    I do not know what is going on over at Dukascopy but they stopped replying to my emails a long time ago on this issue.

    There is not much anyone can do about it, I just thought an update was due since there was a change.

    LL

    • #40 written by LogicaLucidity September 21, 2012 (4 years ago)

      The above gaps were generated using the newest CSV2FXT Script (v0.38).
      I decided to try an older CSV2FXT Script out of curiosity.
      I used v0.35 on the same newly pulled data.
      It produced only the same old 6 day gap we are use to.

      Birt, do you have any thoughts on what is happening?

      LL

      • #41 written by birt September 21, 2012 (4 years ago)

        Quote from the changelog:

        “18.09.2012 โ€“ CSV2FXT v0.38 uploaded. Changes: minor improvement in gap detection, it will now print messages for gaps larger than 2 hours.”

        • #42 written by LogicaLucidity September 21, 2012 (4 years ago)

          *FacePalm*

          Well, that’s embarrassing.

          Thank you Birt.
          LL

        • #43 written by rajiv May 28, 2013 (3 years ago)

          birt i always ignore these gaps, i thought these are sunday and satarday,
          m i correct?

          • #44 written by birt May 28, 2013 (3 years ago)

            No. The gap start and duration is specified by the error message.

  • #45 written by Zack September 30, 2013 (3 years ago)

    I have the same gaps error message. The question is, is the data still usable for backtesting? I don’t see much harm in some hours or days missing, as long as the gaps don’t mess with trades.

    • #46 written by birt September 30, 2013 (3 years ago)

      Compared to many other tick data sources, Dukascopy is very good when it comes to gaps. I disregard them in my backtesting… Such a gap is exactly like having a downtime on your VPS.

  • #47 written by Sven March 31, 2014 (2 years ago)

    Dukascopy download of tick data with Jforex doesn’t work anymore. Anyone recognized this??

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <s> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top