Downloading Dukascopy tick data with JForex

Begin by registering a demo account with Dukascopy and start the JForex platform (you can of course register a live account, the data process is the same). Login using the data in the email you received (note that you don’t need the account ID to login) then go to the Tools menu and select Historical Data Manager. In the lower part of the window, the Historical Data Manager interface should appear; from now on, everything you have to do happens in that part of the window so you might want to enlarge it a bit. Proceed as follows:

  1. Select , (comma) in the Delimiter field. Don’t leave that field blank and don’t select the dot (.).
  2. For the Data Type field, select Ticks.
  3. In the Instrument pane, select all the symbols that you want to download the data for.
  4. Select the Date From and Date To of your choice. Note that the earliest date available for most major currency pairs is 2007.03.01.
  5. It’s safe to leave the Date Format field unchanged.
  6. If you want the data to be exported to a different location, click the Browse button.
  7. Hit the Start and patiently wait until the progress indicator slowly (exactly how slowly depends on the amount of data you selected) crawls to 100%.
  8. Find the CSV files in the location of your choice.

Assuming everything went fine, you’re ready to proceed with preparing your tick data for Metatrader 4.

Note: JForex caches the data to your disk. If for some reason you intend to delete it, it can be found in C:\Users\[your username]AppData\Local\JForex\.cache on Windows 7. On XP/Vista, dig around in your user folder, it should be in a similar path but in Application Data instead of AppData.

  • #1 written by Jim February 14, 2012 (3 months ago)

    Many thanks Birt

    Reply Quote
  • #2 written by Robin February 23, 2012 (2 months ago)

    May I know what’s the filesize for one of these tick data CSV’s for one pair? It’s been 15 minutes since I started downloading and it’s still at 0%… Is the CSV like a few GBs big?

    Reply Quote
    • #3 written by birt February 23, 2012 (2 months ago)

      I wasn’t kidding when I said “crawls”. For example, the CSV for EURUSD for 2007-2012 is over 7 GB, but the size of the download should be much smaller because the files are compressed.

      Reply Quote
  • #4 written by LogicaLucidity March 28, 2012 (1 month ago)

    I have been using this method for several months and have not had any issues thus far. All the credit and thanks goes to Brit. I recently pulled GBPUSD data from Jan 09 – Jan 12, data that I have pulled in the past. This was the first time I used your new CSV2FXT script. I received an alert stating ‘Possible error: Large gap after 2009.06.12 20:59:53 (6.0 days). I have never received one of these before and assume they are new to the script. After looking back at previous back tests using this same time period I noticed that those 6 days were there before. Assuming a mistake, I pulled the data again and used the CSV2FXT with the same result. I then moved to a new PC and made sure to clear the cache located in C:\Users\[your username]AppData\Local\JForex\.cache and cleaned up java. I continue to be missing those 6 days no matter the way I approach it even though I had them before. I then decided to try the Dukascopy historical data page and find that no matter the PC I am on the downloading bar freezes at 8%. If anyone has any idea how this is possible please speak up. Thank you.
    LL

    Reply Quote
  • #5 written by L. April 9, 2012 (1 month ago)

    Hi!

    I’ve tried to dowload tick data according your instructions but the “start” button is grey and I can’t click on it. Can someone tell me what could be wrong?

    Reply Quote
    • #6 written by birt April 9, 2012 (1 month ago)

      Make sure that you configured everything (start & end date, currency etc.).

      Reply Quote
  • #7 written by LogicaLucidity April 9, 2012 (1 month ago)

    To elaborate on my post above… When running the CVS2FXT Script on EURUSD Data from April 1 2007 to April 1 2012 I get the exact same gap, ‘Possible error: Large gap after 2009.06.12 20:59:53 (6.0 days)’. After talking to several people they too are confirming this gap. It has been suggested that the data from JForex was altered after they changed the tick format (was .bin before, now it’s .bi5). I do not know if that was the actual cause. I have yet to determine how many Pairs share this exact 6 day gap. At this point I assume all pairs.

    Reply Quote
    • #8 written by birt April 9, 2012 (1 month ago)

      I’ve just replied to one of the TDS customers who experienced the same problem. It looks like that week was “lost in translation” when they converted the files from .bin to .bi5. I still have old bin files on my Linux box up to January 2012 and I was able to verify that I have the data for that particular week but that doesn’t really do much good.

      I would suggest verifying whether that particular week is missing from any other currency pair and emailing Dukascopy about it. If they truly lost those bin files (or any bin files at all for that matter), they could contact me and I could provide the missing data for any of their currency pairs.

      Reply Quote
      • #9 written by LogicaLucidity April 9, 2012 (1 month ago)

        I will pull data from April 1 2007 to April 1 2012 forming a fresh Cache. I will run CVS2FXT on each one confirming whether or not this gap is universal. I will take my findings to Dukascopy and await a reply from them. Thank you Birt.

        Reply Quote
        • #10 written by birt April 9, 2012 (1 month ago)

          You don’t need to run CSV2FXT or even export to CSV files to check if the gap is there. You can simply browse your JForex cache folder and check if the files exist or not. Pay attention to the file structure – if it matches the structure on the website, January is month 0 not 1.

          Reply Quote
  • #11 written by LogicaLucidity April 9, 2012 (1 month ago)

    If I were only looking for that single common hole you would have saved me a lot of work. Running the CSV2FXT script will enable me to verify that there are not any other large holes that certain pairs share that have appeared in the data since the format change. I want to have a complete picture before I contact them to avoid going to them twice. It is more work, but if I find a second common hole among other pairs I will be glad I did it.

    Reply Quote
    • #12 written by birt April 9, 2012 (1 month ago)

      Someone just told me via email that there is a hole at the same point in the AUDUSD data.

      Reply Quote
    • #13 written by birt April 9, 2012 (1 month ago)

      If you contact them, you can also let them know that they have some big problems on 01.04.2007 (iirc) in their USDJPY and EURJPY data.

      Reply Quote
  • #14 written by LogicaLucidity April 22, 2012 (3 weeks ago)

    Hello Birt,
    You will have to be more specific about the ‘big problems on 01.04.2007 (iirc) in their USDJPY and EURJPY data.’
    I did not find any.

    I am hoping that you might confirm that this 6 day hole I am getting with every pair was not there before the format change.

    Is there anyway you can do that? You are the only one I have talked to who has a cache prior the format change.

    You are free to email me.

    I pulled all data for all 22 pairs for as far back as possible for each up to April 1, 2012. (90+GB)

    I ran your CVS2FXT script on each one.

    Every pair that has data available during June of 2009 has the exact same 6 day gap.
    Other pairs suffer from Gaps but none that are in common.

    This 6 day gap was not there before the format change on several of the pairs.
    I can only assume that it was not there on the rest as well.

    I will contact Dukascopy and inform them of the error and hope for a response.

    Here are the results:

    AUD/CAD Date Span:(2010.02.16-2012.04.01)
    No Gaps

    AUD/JPY Date Span:(2007.03.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    AUD/NZD Date Span:(2008.12.22-2012.04.01)
    large gap after 2008.12.22 16:25:03 (15.0 days).
    large gap after 2009.06.12 20:58:35 (6.0 days).

    AUD/USD Date Span:(2007.03.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    CAD/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:51 (6.0 days).

    CHF/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    EUR/AUD Date Span:(2007.3.30-2012.04.01)
    large gap after 2007.06.01 20:59:36 (24.0 days).
    large gap after 2007.06.26 08:03:43 (19.0 days).
    large gap after 2009.06.12 20:59:38 (6.0 days).

    EUR/CAD Date Span:(2008.09.23-2012.04.01)
    large gap after 2009.06.12 20:59:38 (6.0 days).

    EUR/CHF Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:47 (6.0 days).

    EUR/GBP Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:53 (6.0 days).

    EUR/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    EUR/USD Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    GBP/CHF Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:39 (6.0 days).

    GBP/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:53 (6.0 days).

    GBP/USD Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:53 (6.0 days).

    NZD/USD Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:48 (6.0 days).

    USD/CAD Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:52 (6.0 days).

    USD/CHF Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:47 (6.0 days).

    USD/HKD Date Span:(2010.10.15-2012.04.01)
    large gap after 2010.11.10 16:33:01 (158.0 days).

    USD/JPY Date Span:(2007.3.30-2012.04.01)
    large gap after 2009.06.12 20:59:51 (6.0 days).

    USD/MXN Date Span:(2010.10.15-2012.04.01)
    Pair not availible on my HoTForex Demo MT4 (409) platform.

    USD/SGD Date Span:(2008.09.28-2012.04.01)
    large gap after 2008.09.29 07:10:36 (6.0 days).
    large gap after 2008.10.06 16:35:24 (13.0 days).
    large gap after 2008.11.03 15:37:51 (647.0 days).

    LL

    Reply Quote
    • #15 written by birt April 22, 2012 (3 weeks ago)

      Those files have a size of 0 on the Dukascopy site, both the new format (bi5) as well as the old one (bin). In my download cache, they are fine. I will contact Dukascopy and offer to send them the files if they need them.

      Reply Quote
      • #16 written by LogicaLucidity April 22, 2012 (3 weeks ago)

        I have contacted them as well detailing out all I know.
        Hopefully they will reply.
        Thank you Birt.
        LL

        Reply Quote
        • #17 written by LogicaLucidity April 25, 2012 (3 weeks ago)

          Dukascopy’s Response to my concerns:

          Greetings,

          We do not provide any services for data download in a BIN or BI5 format.

          Best regards,
          Dukascopy Support

          I replied asking them to reconsider… explaining it in even greater detail.

          There is only so much one can do.

          Reply Quote
          • #18 written by birt April 26, 2012 (3 weeks ago)

            Well, at least you got a reply… I didn’t get one yet.

            Did you tell them that you downloaded the data using their JForex client? Also, did you tell them that downloading the data for that period via the website doesn’t work either?

            Reply Quote
            • #19 written by LogicaLucidity April 26, 2012 (2 weeks ago)

              Yes and yes. I went into elaborate detail.

              I provided links to their own site where the JForex client is presented.

              I really do not know what steps to take now.

              We have the facts. We just need someone to listen.

              Reply Quote
  • #20 written by Future April 30, 2012 (2 weeks ago)

    i’v got the same problem on EUR/USD H1

    2012.04.30 17:59:01 CSV2FXT EURUSD,H1: Alert: Possible error: large gap after 2009.06.12 20:59:48 (6.0 days).

    Reply Quote
    • #21 written by birt April 30, 2012 (2 weeks ago)

      As stated above, those files are broken (0-sized) on the Dukascopy website.

      Reply Quote
  • #22 written by LogicaLucidity May 4, 2012 (1 week ago)

    BOOMSHAKALAKA!!!
    My further attempts have finally paid off!

    Dukascopy’s most recent response:

    Greetings,

    Thak you for the reporting the problem!

    There are no any connections with a switching from BIN to BI5 format.
    In any case, the data will be restored as soon as possible.

    Best regards,
    Dukascopy Support

    Reply Quote
    • #23 written by LogicaLucidity May 8, 2012 (1 week ago)

      Here is their reply about when things will be fixed and if Gold and Silver will be restored…

      Greetings,

      We are going to repair all data we provide.
      It has to be done next week.

      Best regards,
      Dukascopy Support

      I Hope this is the end of this.

      Thank you Birt for all you have done,
      LL

      Reply Quote
  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top