How to download free tick data

Nowadays there are many sources of tick data and this guide attempts to cover some (well, most) of the popular providers where you can download free tick data. You may use the navigational links here to jump directly to your preferred provider or you can read the whole article and decide which source is best for you. Before you ask, I favor Dukascopy due to the data quality and accessibility.

• Dukascopy
• Integral & Pepperstone
• Oanda
• MB Trading
• HistData.com
• Gain Capital (aka Forex.com)

Dukascopy

Background

Dukascopy is universally acclaimed as one of the best brokers nowadays. Personally, I’ve never had a live account with them so I cannot confirm or deny that, but many people classify it as “the best ECN broker” and I have yet to read bad things about them. From my point of view, their major drawback is the lack of a MT4 client (it’s possible to run MT4 EAs, but very tricky). However, if you’re looking for a really solid ECN broker and Metatrader expert advisors are not your kind of cookie, look no further. Their JForex client supports extensive automation but you definitely won’t find as many commercial automated trading systems for that as you can find for Metatrader 4.

By far the most popular tick data source, Dukascopy offers free data spanning from April 2007 (for most currencies) to now. Despite being offered at no cost, it’s commercial-quality data and it actually approaches the 99% quality displayed in the strategy tests (which, by the way, is just a number in the FXT header). From what I can tell, it is updated hourly and the source is none other than the Dukascopy data feed server. Bottom line is: I couldn’t wish for a better data quality.

Show available currency pairs »

Acquiring

The most convenient method to download from Dukascopy is via the Dukascopy historical data page which features a web application that lets you get the tick data for a specific time range. You will be prompted to select the date format – you may choose whatever format you want for the date, the FXT conversion script will be able to autodetect and parse it.

Some other options for downloading are:

• By means of JForex, the Dukascopy client. Please see the Downloading Dukascopy tick data with the JForex client guide for more details.
• Using my PHP script collection. See the Downloading and parsing the Dukascopy tick data with Birt’s PHP scripts guide.
  Warning: this is for the tech-savvy people that like to be in full control of the tick data process.
• Using the Dukascopier (available for download on the tick data downloads page), an application wrote by Tim Cragg. If you don’t already have the .NET Framework 4.0 installed, you need to download and install it – a link is available in the Runtimes section of the same tick data downloads page. Please note that I do not provide any support for this; if you run into any problems with it, you should email Tim, whose address is in the application About tab. Caution: currently, using the Dukascopier you cannot download any data past January 2012.
  The advantage of using this method is that you can export the CSV while downloading, whereas the PHP scripts take several hours to do that.

No matter which method you use, please take the time to read the “Purpose” and “No warranty by Dukascopy” sections on the Dukascopy tick data page.

Pros & cons

Pros:
- multiple download methods
- extensive time interval coverage
- wide array of available pairs
- very good quality
Cons:
- up to November 2010, the data was not top-of-the-order-book, but rather aggregated in 0.5 pip increments
- the data for some pairs (minors & metals) doesn’t go as far back as 2007

Pepperstone and Integral

Background

Pepperstone is an ECN broker founded in 2010, with a solid reputation and a growing client base. I never had an account with them, but I heard good things from many people that have one and I have yet to hear from an unsatisfied client. The tick data they are offering is gathered by their partner Integral, one of the leading liquidity providers. The quality is very high, with spreads lower than the Dukascopy in places.

The oldest data available is from 2009 and the available range of currency pairs is mostly limited to the majors (there are not many minor currency pairs and no metals). The data does not include the volume.

Show available currency pairs »

Acquiring

The data can be downloaded at the Pepperstone tick data page or at the TrueFX downloads page, with the latter requiring registration (which is free).

The data is split in files that span one month. To use it, go through the following steps:

1. Download the months of interest (should be consecutive).
2. Unpack all the zip files.
3. Concatenate the files in order. The easiest way to do that is:
   1. Fire up a command prompt (Start->Run->cmd.exe)
   2. Change the directory to where you unpacked the files into (e.g. cd "\Documents and Settings\Administrator\My Documents\Downloads\")
   3. Use the type command (e.g. type EURUSD-2011-01.csv EURUSD-2011-02.csv EURUSD-2011-03.csv EURUSD-2011-04.csv > EURUSD.csv)

If you need more detailed instructions for concatenating files, please use a search engine. As a side note, if you’re using Firefox you might want to get the DownThemAll addon – it can download all the data for the pair(s) you want without too much clicking.

Pros & cons

Pros:
- very good quality
Cons:
- the first available date is 2009
- the data is split into months and the files have to be downloaded individually and manually concatenated (merged)
- the pair name is included for every tick which is a complete waste of space and bandwidth

Oanda

Background

Oanda is one of the oldest brokers around, the firm being established in 1996. Their Forex trading service with their proprietary trading platform became available in 2001 – I distinctly recall trading their demo on 9/11 and watching the news. Nowadays, Oanda is an US-based market maker broker with a very good reputation. Recently, they also added a MT4 platform but to my dismay, the lot size was 100k which effectively threw out the biggest advantage of their platform which was the ability to trade any sum you wanted (even $1!). One of the other things they’re quite famous for is their interest rates which are very competitive and are calculated by the second: you don’t see swap in the fxTrade platform; the interest is simply calculated when the trade ends.

Since Oanda is a market maker, their data naturally has less ticks than that of an ECN which may or may not be good depending what you want. The Oanda tick data is unique in its time span: it starts being available as early as the start of 2004. Unfortunately, the currency pairs available are limited to 5 majors and downloading the tick data is restricted to users that have an Oanda account with a balance of at least $1000.

Even though probably pretty much nobody cares, it’s worth mentioning that the tick data offered does not include the volume which is quite normal seeing that it comes from a market maker.

Show available currency pairs »

Acquiring

Before you try to download, I’ll stress this once more: the tick data is completely free but you need to hold an account with Oanda that has at least $1000 (there is an academic exemption, though).

Downloading the data is queue-based. You head to the Oanda historical rates page, then select Make an order which takes you to the tick data order form where you can select the start and end date, the currency pair and the output format; for the latter, everything should do beside XML. When ready, hit Place your order which puts your request in the queue and emails you once processing is done which typically takes from several hours to several days. Note that you can’t have more than one request queued at a time. Also note that you cannot request data for the past two months, in other words the ending date in your request form must be set to at least two months ago.

When your file is ready, you just download it and that’s all.

Pros & cons

Pros:
- data as old as 2004 is available
- easy to acquire
Cons:
- an account funded with at least $1000 is required
- only 5 major pairs are available
- the whole process is queue-based and access to the file is not instant
- the most recent data cannot be requested, the tick data ending date is limited to 2 months before the current date

MB Trading

Background

Another US-based broker, MB Trading provides a wide array of products; when it comes to Forex, ECN services are offered to their clients. Together with the Desktop Pro proprietary trading platform, Metatrader 4 is available but with a lot of peculiarities such as a 10k lotsize, the requirement for a special authentication key, lots of partial fills, no expiration on orders and a daily server restart that takes some 5 minutes at 5 PM US EST. These things aside, their spreads are very good and the commissions are decent as well.

The tick data offered is top-of-the-order-book without volume and it is available for a lot of pairs: the selection box includes all majors and some minors, their count almost rivaling the number of pairs available at Dukascopy. The data is completely free, but unless you already have an account with MB Trading, you are going to have to sign up for a demo account (which is of course also free). The earliest available date is January 2011.

Show available currency pairs »

Acquiring

Head to the MB Trading tick history download page. Log in using your account credentials (if you don’t have an account, just open a demo account, it’s free). If you got lost in the registration process, once you log in to your account, the page can be accessed by clicking the Tick history link in the Tools menu.

Once you get to the actual download page, you will have to select the currency pair of your choice then start browsing and enabling each day that you want to download. Note: if you want to actually do something with the data, make sure you’re getting consecutive days, not one random day here and there.

When you’re done selecting each and every day that interests you (quite a lot of clicking if you want all of them), click Download and pray that it actually works. When I tried this, I ran into a nasty problem that would stop the download before it was finished and I had to download each day separately. Hopefully, it was just an issue on my end.

Assuming you did everything I suggested above, you should end up with a file named Tick????.zip. You have to unpack it and inside you will find a bunch more zip archives, one for each day. You also have to unpack those and when that’s done, the last step to do before turning them into an FXT is concatenating them in order (see Pepperstone & Integral above for more info regarding the concatenation).

Pros & cons

Pros:
- very good quality
- wide array of available pairs
Cons:
- the data is available only starting with 2011
- the data is split into days and the files have to be unpacked and manually concatenated (merged)
- the pair name is included for every tick which is a complete waste of space and bandwidth
- the download process is quite unfriendly
- concatenating the files for each day can be quite a pain in the ass if you’re doing it for a time period spanning several months

HistData.com

Background

A project by a group of traders that are making the data available for free at their own expense – an initiative that I applaud, there’s no hidden interest whatsoever.

The availability date varies by symbol and goes as far back as 2000 for some majors. However, the data is not very consistent – I downloaded two samples of EURUSD tick data, one from March 2011 and one from January 2012 (the reason being that I needed to figure out the GMT & DST); as it turns out, the March 2011 file has quotes from a market maker broker with 4 digits and a fixed EURUSD spread of 3 pips with a GMT offset of 4 (I think), while the January 2012 file has quotes from what appears to be an ECN broker (possibly MB Trading) with a GMT offset of -5. As such, the CSV2FXT script will prompt you to enter the GMT and DST offset manually.

As a nice bonus feature, the archives include text files that specify all gaps larger than 60 seconds.

Show available currency pairs »

Acquiring

Head to the HistData.com and start downloading. Make sure you’re getting the Generic ASCII tick data. You can even do it via FTP if that’s more comfortable for you.

If you wish to use several of the downloaded files in your CSV, you have to unpack and concatenate them in order (see Pepperstone & Integral above for more info regarding the concatenation).

Pros & cons

Pros:
- very wide array of available symbols
- gap info
- easy to download
- good date availability range for many symbols
Cons:
- the data source is inconsistent (4 digit, fixed spread MM mixed with ECN)
- the GMT offset is inconsistent (and perhaps the DST as well)

Gain Capital aka forex.com

Background

Gain Capital is a rather old (founded in 1999) market maker broker that managed to get somewhat of a bad reputation. They hit their head on a half a mil fine that was graciously awarded to them in the year of our lord 2010 by the NFA for bad some bad practices, and I quote: “…engaged in abusive margin, liquidation and price slippage practices that benefited Gain to the detriment of its customers”. Anyway, I’m not pitying them since their forex.com domain name rod must be bringing them a ton of unsuspecting fish anyway.

Aside from offering shady trading conditions, they are also offering a selection of tick data, but last time I checked, its quality was just as poor. I can’t seem to remember what’s the first year that has tick data available and I can’t be bothered to download it again and find it out. Once more, fair warning: the data quality is pretty bad – not only is the data missing large chunks of time, but it’s also sporting misaligned periods (e.g. you can sometimes find data from the previous week in the file for the current week). The only good thing about it is the fact that a lot of symbols are available: from currency pairs major, minor and exotic to metals and various stocks and indices.

Show available currency pairs »

Acquiring

If despite my warnings above you’re feeling a bit masochistic and want to play with it, you can head out to the Gain Capital rate data page and fetch the archives; I recommend using a download manager.

Before finding other tick data sources, I used to use this data for testing so I perfected some scripts for it, which are still available for download as a PHP script collection on the tick data downloads page in the Miscellaneous section.
Inside you will find:

• A script to process the data into a CSV. This script is heavily based on the original by carloss from the indo-investasi forum.
• A script to somewhat filter & sort the resulted CSV.

Since these are totally obsolete to me, I’m not going to provide more details and you’re going to have to figure out how they work on your own. Shouldn’t be too hard, though. Please note that I’m not volunteering any support for these scripts.

Pros & cons

Pros:
- extensive array of available symbols including stocks & indices
Cons:
- the download process is cumbersome
- the data is split into weeks and the files have to be unpacked and manually concatenated (merged)
- the data quality is poor, there are gaps and misalignments
- there are no up to date tools to process the data