How to prepare your tick data for Metatrader 4

If you’re reading this guide, I will assume you already downloaded your tick data either from one of the free sources mentioned on the Downloading free tick data page or from your own private source and now you need to use it in Metatrader 4.

Note: please refer to the old version of the guide if you’re using MT4 build 509 or older.

Converting the data

To put it simply, Metatrader 4 does not know how to directly read a CSV file containing tick data and thus it cannot use it in your backtests. However, what it can read is a proprietary file format that contains ticks so all we have to do is convert from our CSV file to FXT, the latter being the format I mentioned. To this end, I wrote a few scripts but later decided to make it less complicated and merged them into a single script that should be able to convert data from pretty much any source and put it into an FXT. One thing to note is that during backtesting, Metatrader 4 also makes use of the HST files when calculating the indicators so in order to have accurate tick data backtesting, you must also have HST files that match your FXT file.

So, what you need to convert is the CSV2FXT binaries that can be found in the tick data downloads section. It’s important to note that for the newer MT4 builds you will need CSV2FXT v0.44 or higher. Metaquotes changed the FXT and HST format after build 509 so all builds newer than that require the modified script. It’s a good idea to get the latest script even if you already have a compatible version.

Here’s a short guide on how to convert the tick data to an FXT file and a series of HST files:

  1. If you haven’t done so already, you have to head to the tick data downloads section and download the CSV2FXT binaries archive.
  2. Copy the files from the zip archive to your MT4 data folder. To find out what is your data folder, open MT4, head to File and click Open Data Folder, which will open an explorer window with your MT4 data folder (typically looks like C:\Users\[username]\AppData\Roaming\MetaQuotes\Terminal\[32_character_hex_string]\). There’s a directory structure inside, make sure the files land in the proper places (CSV2FXT.mq4 and CSV2FXT.ex4 must be in MQL4\Scripts, FXTHeader.mqh must be in MQL4\include, CsvReader.dll must be in MQL4\Libraries). Note that if you are using the /portable switch, your data folder will be the same as the MT4 installation folder.
  3. Move the tick data file (the CSV file) to MQL4\Files in the same MT4 data folder.
  4. Open a chart for the pair that you have data for (if you have an EURUSD.csv file, you MUST open an EURUSD chart).
  5. Select the timeframe that you wish to generate the FXT for. For instance, if you want to backtest on M1, then select M1 as the chart timeframe. Please note that the FXT file that you create for a particular timeframe (even M1) will NOT work for any other timeframe; you simply have to generate a new FXT if you want to backtest on another timeframe.
  6. Ensure that your terminal is connected to the broker (look in the lower right corner, if it says “not connected” you need to fix that before proceeding).
  7. Make sure that DLL calls are allowed. If you don’t know how to do that, you have to open the Tools menu, select Options, select Expert Advisors and ensure that Allow DLL imports is enabled while Confirm DLL function calls is disabled.
  8. Double click the CSV2FXT script in the navigation panel (it’s in the scripts section).
  9. Configure the parameters in the window that pops up.
    • CSV2FXT version – this is a parameter that is only meant to give you a quick indication of what version you have installed. Changing it has no effect.
    • CSV filename – you can leave this blank if the file is named just like the symbol and has a CSV extension (e.g. EURUSD.csv); otherwise, just type the file name.
    • Create HST files – this setting must be true in order to create the HST files that you need for your backtest. You can set it to false if you already generated HST files for the symbol with the same GMT/DST settings and you’re just generating an FXT for a different timeframe. You must create new HST files every time you change the GMT or DST.
      Note: Enabling this setting will create HST files for the whole time span of your tick data file regardless of the time range selected.
    • Spread – the fixed spread of your resulting FXT file, expressed in pips (2.3 will result in a 2.3 pips spread). Leaving it set to the default of 0.0 will make the converter use the current spread of your broker. Please pay attention to the fact that many brokers are widening their spreads during the weekends.
      If you intend to use real spread (the variable spread in your CSV), you can leave this parameter set to 0.0. Note: the Spread field in the MT4 backtesting window has no effect when backtesting using tick data.
    • Start date and End date – these fields control the time span of the FXT file. You can leave these fields set to their default values (1907.01.01), in which case the converter will just use the whole time range available in the CSV file.
    • Use real (variable) spread – as its name suggests, enabling this parameter will make your resulting FXT use the real (variable) spread from your CSV file.
      The Tick Data Suite will autodetect whether your FXT is using real spread or not so there’s nothing to worry about if you’re using that. Once more, note that the Spread field in the MT4 backtesting window has no effect when backtesting using tick data.
    • Spread padding – if using real spread, you can pad it by a given number of pips – if you want to pad it by 0.8 pips, just specify 0.8 here.
    • Minimum spread – if any spread is encountered that is lower than the value specified for this parameter, it will be adjusted to this value. This is only applied when using real spread.
    • Commission in pips – if you want your FXT to have a commission, you can configure the desired value here. The figure is round-trip and it is expressed in pips.
    • Commission in account currency – as an alternative to having the commission in pips, you can also set the commission in money. The value is expressed in base account currency per lot round-trip. If you fill this in, any value specified for Commission in pips will be disregarded.
    • Leverage – changes the leverage of your FXT.
    • FXT GMT offset – if you want your FXT to have a GMT offset other than 0, specify it here.
    • FXT DST setting – the DST setting of your resulting FXT – simply select the DST setting that you would like the file to have. Note that the US DST setting calculates the DST according to the regulations that were applied starting with 2007.
    • CSV GMT offset – the GMT offset of the data in your CSV file. The conversion script is able to autodetect the formats of several free tick data providers and apply the correct setting here, so it’s probably safe to leave it set to “autodetect”. If you get a message in the experts log about the script not being able to identify your tick data source, you can set the CSV GMT offset here manually.
    • CSV DST setting – the DST setting of your CSV. Should be safe to leave it to “autodetect”. Otherwise, use the same guidelines as for the FXT DST setting.
    • Time shift – Enabling this parameter will shift all the generated data 28 years in the past. This is intended for use with EAs that are suspect of having hardcoded days for the purpose of cheating backtesting. The reason behind the 28 year shift is that the calendar is identical when it comes to the days of the week and leap years. This is not a foolproof method and some EAs may have legitimate reasons to yield different results when backtested with a shifted time.
    • Price multiplication factor – All the prices will be multiplied by this value. It’s typically safe to leave this parameter set to 1.0 – don’t change that if you’re converting Forex data. Some brokers use an adapted value for CFDs, indices, metals and so on – instead of the normal price (for instance 12.3456) they will have a value multiplied by a certain value (for example 1234.56); in these cases you have to figure out the multiplier by looking at the chart prices versus the CSV prices.
    • Create M1 FXT, Create M5 FXT, Create M15 FXT, Create M30 FXT, Create H1 FXT, Create H4 FXT, Create D1 FXT, Create W1 FXT, Create MN FXT – These parameters are meant to allow you to create multiple FXT files in a single run. By default, the script will create the FXT for the timeframe of the chart you are running it on, no matter if the parameter for that specific timeframe is enabled or not. Only enable these parameters if you actually need the FXT for a different timeframe.
  10. Click “Ok”. Once you do that, the data generation process will start and it will typically take half an hour to several hours, depending on the data range and volume, possibly even more if you’re using a slow machine. A progress indicator will appear in the upper left side of your chart and when the processing is finished, you will get an alert. During the conversion, some data is being printed in the experts log and if you have any problems with the script it’s probably a good idea to keep an eye on that.

Once you’re done with all the steps above and the script finishes processing, you will be asked whether you want the script to move the files to their proper locations. If you choose Yes, restart the terminal to ensure the HST files are properly synchronised and skip to the next guide (Using the tick data in your backtests). If you choose No, you will have an FXT file and a bunch of HST files in your MQL4\Files folder and before actually using them you need to copy them to the correct location. Proceed with the following steps:

  1. Exit the Metatrader 4 terminal.
  2. Move all .HST files from MQL4\Files to history\[your_server_name]\.
    Pay very close attention if you have multiple server directories in your history folder – you will have to move them into the one that’s correct for active account!
  3. Move the generated FXT file from MQL4\Files to tester\history.

At this point, you’re done with the conversion and setup! You’re now ready to proceed to backtesting, but please note that it’s not as easy as clicking Start in the backtesting user interface. You will have to use some additional tools which are described in the Using the tick data in your backtests section.

If you run into any problems, please head to the FAQ & Troubleshooting page.

  • #377 written by Jelks February 7, 2014 (7 months ago)

    Thank you Birt and OneJin. Especially on the advice about autoupdate.

    I initially couldn’t find the “Webinstall” folder, I had to do some googling. Found it. At least on Win7, it’s in C:\ProgramData\MetaQuotes (you may have to enable “show hidden files and folders” in Explorer to be able to see it.

    I renamed that folder to “Webinstall Foo”. But I have to wonder, why just one “Webinstall” folder. I’ve got 2 live accounts, a number of demo accounts, and of course, the Tick Data Suite…

    I don’t know how anybody could do anything backtesting-related without this website LOL!

    Jelks

  • #378 written by OneJin February 8, 2014 (7 months ago)

    Hello!
    What is the workflow to set another currency pair to test on the same terminal? Would the results be correct if I just move another pair .csv file into Files folder, then drag CSV2FXT onto terminal graph (patch it), restart terminal. Or do I have to create new MT and do whole stuff from the very beginning?
    Thanks for advice. :)

    • #379 written by birt February 8, 2014 (7 months ago)

      I’m not sure what you’re talking about when you say “patch it”.

      You don’t need to restart the terminal or anything like that, simply run CSV2FXT on the desired chart. One thing though: due to several considerations, I would not recommend creating FXT for several pairs at the same time in a single MT4 terminal.

  • #380 written by OneJin February 8, 2014 (7 months ago)

    Birt, imagine the following… I have successfully tested EURUSD. And now I want to test GBPUSD. Do I need to delete terminal with EURUSD and follow this tutorial from the very beginning (create MT, copy Experts from here into my MT Experts, copy GBPUSD into terminal files, run CSV2FXT on desired timeframe, restart terminal, start testing) ?
    Or I can copy GBPUSD .csv file into my MT Files, run CSV2FXT on GBPUSD H4 ( I test H4), restart terminal and start testing?

    • #381 written by birt February 8, 2014 (7 months ago)

      The latter – you don’t need to delete anything. Simply run CSV2FXT on GBPUSD, restart terminal, start testing.

      • #382 written by OneJin February 8, 2014 (7 months ago)

        :) Got it. Thank you, Birt!

  • #383 written by OneJin February 8, 2014 (7 months ago)

    Sh-h-h-hit…
    There is another issue. Tester is testing, but there is no output in Optimization results tab. What can it be ?

    • #384 written by OneJin February 8, 2014 (7 months ago)

      My build is 604

    • #385 written by OneJin February 8, 2014 (7 months ago)

      Oh-h-h-h… There is new version – Birt’s CSV2FXT script v0.45 binaries. But I have downloaded 0.44 few days ago. Can it be the reason? And one more thing – do I have to follow your download section and download new versions from there all the time?

      • #386 written by birt February 8, 2014 (7 months ago)

        That’s not the problem for sure. v0.45 simply adds some error messages in case someone is using the wrong CsvReader.dll.

        In all likelihood, there are no results because negative results are hidden by default. Right click somewhere in the optimization tab and disable the “hide useless results” option (or whatever its name was).

        If you want to keep up to date with the tick data updates, you can subscribe to the RSS of http://eareview.net/tick-data/changelog (alternatively you can subscribe to the comments by email). Anyway, there’s no need to update every single time something is posted there; for instance if CSV2FXT 0.44 is working fine for you, there will be no improvement with CSV2FXT 0.45.

    • #387 written by OneJin February 8, 2014 (7 months ago)

      Birt, thank you again and again :) !!!
      You were right there areresults, but not profit.

  • #388 written by OneJin February 9, 2014 (7 months ago)

    Hello!
    I want to include Math library into my EA. But there is no such library in newly setupped MT. Can you tell me where can I found librares to download? I would be very grateful!

    • #389 written by birt February 9, 2014 (7 months ago)

      That’s a question best suited for the MQL4 forum. I can’t provide MT4 support.

  • #390 written by Jelks February 10, 2014 (7 months ago)

    Birt, I downloaded a demo copy of build 604 and ran your 0.46 CSV2FXT on the same USDCHF.csv that I had done with the TDS-enabled build 509. I then ran the same 4-month backtest I had gotten with 509. The 4-month result was only $8,057.84 as opposed to the old-build TDS-enabled Tester which yielded $80,824.54! Then I realized, “duh, it’s using broker information (the reported spread used was 15!) instead of FXT information.”

    So can you have 2 different Tick Data Suites, one for 509 and one for 604?

    Thanks,

    Jelks

    • #391 written by birt February 10, 2014 (7 months ago)

      Tick Data Suite v1.3.0 will work with build 509 as well as with build 604 so if you want you can have different MT4 installations with different versions. You can even have build 225 if that helps in any way. Just make sure you use the right CSV2FXT and the right FXT & HST files in each of them.

      • #392 written by Jelks February 11, 2014 (7 months ago)

        Birt, I ran the TDS Autoupdate just to make sure I had the latest version and it said there was nothing to update. Then I ran the TDS Configuration utility and applied it to my build 604 installation. It gave me this error message:

        ———–
        You are using a Metatrader 4 build that is not supported by your current Tick Data Suite version. Please check the website for a newer version. If your MT4 build was recently released, please allow up to a week for a new Tick Data Suite version to be posted.
        ———–

        Does not the TDS Autoupdate always update you to the lastest version of the TDS? And how do you tell which version of the TDS you have?

        Jelks

        • #393 written by birt February 11, 2014 (7 months ago)

          Autoupdate does not update your installation to beta versions. TDS v1.3.0 is still beta and consequently it was not automatically downloaded. I’m probably going to release the stable build in a few days as there was a single bug reported and it was fixed right after the beta release.

          For now, you have to download and install it manually.

  • #394 written by Jon February 24, 2014 (6 months ago)

    There are still a few brokers who use 4 digit pricing. Is it possible for your csv2fxt script to have a feature where you can implement 4 digits or rounding up/dn x digits? That would allow users to import the data and still see how their strategy would work when whole pips are used (vs fractional pips).

    It may not be enough to simply used fixed spreads, as the fractional pips may skew granularity that is lost when price movements only move whole pips.

    • #395 written by birt February 24, 2014 (6 months ago)

      CSV2FXT should work fine with 4 digit prices, although I haven’t tested this functionality in a very long time.

      • #396 written by Jon February 24, 2014 (6 months ago)

        I men actually converting dukascopy data (5 digits) into 4 digits. I’m sorry if this wasn’t clear earlier.

        • #397 written by birt February 24, 2014 (6 months ago)

          If you run CSV2FXT on a 4 digit broker, a 4 digit FXT will come out no matter the source of the tick data.

          If you want to convert the CSV file to another CSV with 4 digit prices, that’s something else entirely and you’ll probably have to write some code of your own for it.

          • #398 written by Jon February 24, 2014 (6 months ago)

            So in other words, there is no need to have any special code to get 4 digit tick data specifically for mt4 broker as long as I am using a 4 digit broker/account when running the csv2fxt script?

            How does csv2fxt handle the rounding in this case? If the tick bid shows 1.23148, will the fxt file store 1.2315 or 1.2314?

            • #399 written by birt February 24, 2014 (6 months ago)

              If I remember correctly, in the FXT file the whole 1.23148 price will be stored but since the number of digits will be set to 4, MT4 will interpret that number as it sees fit (I’m guessing it’ll round up to 1.2315).

              • #400 written by Jon February 25, 2014 (6 months ago)

                Ok. This makes sense. I would like to test that….if only there was a ‘tick printer’ type of script that could do this post b509…..

                I’m wondering if the broker themselves can still send the feed as 5 pips, but then on the mt4 server/client’s computer it is 4 pips and the rounding is done automatically?

  • #401 written by Jon February 25, 2014 (6 months ago)

    I had a question about the minimum required fields for preparing the tick data when it is in csv format:
    {Timestamp:yyyy.MM.dd HH:mm:ss.fff},{BidPrice},{AskPrice}
    or timestamp, bid price, ask price is all that is needed, correct? The bid and/or ask volumes are not needed, correct?

  • #404 written by Jon February 25, 2014 (6 months ago)

    I also had a question in regards to the commission. If no commission is specified, does the backtester use the broker’s specified commission for the pair? Or is the entire [real spread] backtest ran with 0 commission?

    This is for brokers who charge a separate commission from their spread.

    • #405 written by birt February 25, 2014 (6 months ago)

      If no commission is specified, the FXT commission will be 0 regardless of what the broker is charging. The reason is that you can’t find out the order commission until you open and close a position.

      • #406 written by Jon March 11, 2014 (6 months ago)

        So where is the commission shown on the report? Do you have a screenshot example of this? I cannot find the commissions deduction in the trade list or report. How would I verify that the commissions are being taken correctly for each trade?

        • #407 written by birt March 12, 2014 (6 months ago)

          The commission is not printed anywhere on the report.

          To verify that it was processed, perform some calculations for a trade: multiply the number of pips by the pip value and subtract the commission, it should yield the profit/loss of the trade.

  • #408 written by Jozo February 26, 2014 (6 months ago)

    Problems!
    The last time I run a Tickdata back test on EURUSD about 3 weeks ago, all went well. This was before the MT4 updates. Now tried to run a backtest, the results are totally different with modelling quality of 25%. Has anyone experienced this? Where else should I be looking or what should I be doing to rectify the situation. Should I start from the beginning with new data download?
    Thanks

    • #409 written by birt February 26, 2014 (6 months ago)

      You don’t need to download the data again, but if you updated from 509 to 600+ you will need to recreate the FXT and HST files.

      You need the latest version of Tick Data Suite (1.3.1 at this time) and you MUST also upgrade CSV2FXT to v0.46. The new builds have a new HST and a new FXT format and consequently the older CSV2FXT scripts will not produce proper results.

      What you need to pay a lot of attention to is the new folder structure. Basically everything is now stored in the MQL4 folder, likely somewhere in your AppData\Roaming folder. You can go to File -> Open Data Folder to find out where. You have to put CSV2FXT in the MQL4 folder just like it is stored in the archive. You have to place the CSV file in MQL4\Files.

      There’s also the option of using the /portable switch to allow using the data in the MT4 installation folder; look it up in the MQL4 forum for more info if you’re interested.

  • #410 written by Jozo February 27, 2014 (6 months ago)

    Thank you very much….now working as intended!

  • #411 written by jivetrader March 5, 2014 (6 months ago)

    Step 6 says ‘Ensure that your terminal is connected to the broker’

    i am concerned that if i login to my brokerage account then MT4 will append/overwrite my nice clean tick data with M1 broker data. (i am using TrueFX tick data.) please advise, thank you.

    • #412 written by birt March 5, 2014 (6 months ago)

      MT4 will not append/overwrite your tick data CSV. It doesn’t even know the format. Also, it will not append or replace the data that is written in your FXT or HST files by the CSV2FXT script.

      Overwriting when connecting to the broker happens only for HST files that are placed in the correct broker folder (so after they are generated) and only for the currency pairs and timeframes that have open charts in your terminal. You can prevent this by simply not logging in with that terminal or by setting the HST files readonly. Method 1 (not logging in) is more efficient.

      In essence: you have to be logged in when you create the tick data files (CSV2FXT) but you can be logged out when backtesting with tick data. You could easily accomplish this by creating your FXT files with a terminal and backtesting with another.

      • #413 written by jivetrader March 5, 2014 (6 months ago)

        thank you birt. a bit confused by paragraph 2 but i understand that the tick data is safe in its own folder, and that MT4 can’t read it thus can’t display it or use it.

        in paragraph 3 i think you are saying, use 1 (connected) platform to do the data converting while simultaneously starting your backtesting on another MT4 install. this implies moving the files over from the connected platform to the non-connected, as they are ready? i am new here so please forgive the ignorance

        • #414 written by birt March 5, 2014 (6 months ago)

          You got the idea but you don’t have to use them simultaneously. Simply use a platform that is connected to the broker to generate the FXT and HST files with CSV2FXT; at the end of the process, answer no when asked to move them and instead manually move them to another terminal that does not connect to the broker (e.g. you can delete the account to achieve this). Once that’s done, start the 2nd terminal and start backtesting.

          In any case, even if you don’t do the above, only recent data in HST files is affected so it will affect your backtests ONLY if you’re using indicators on timeframes other than the backtest timeframe AND if you’re backtesting recent periods (how recent depends on your timeframe). Chances are your backtests wouldn’t be affected by this anyway.

  • #415 written by jivetrader March 5, 2014 (6 months ago)

    Possible error: gap after 2009.12.25 06:30:32 (62.0 hours).

    …meaning there is no data for a few days due to the holiday, and i should not be concerned, correct?

    however i also see a gap of 720.0 hours. what is recommended to do ?
    TIA

  • #416 written by jivetrader March 5, 2014 (6 months ago)

    re: last script parameter from the instructions above:
    CreateM1, CreateM5, CreateM15, CreateM30, CreateH1, CreateH4, CreateD1, CreateW1, CreateMN etc.

    i don’t understand the description. please verify- selecting true for each TF will output FXT files but not HST files for these TFs. thus if u need multiple HST files in different TFs, u still must run the script with the chart set to the desired TF. setting these true only helps if you need FXT files only, correct?

  • #417 written by jivetrader March 5, 2014 (6 months ago)

    Birt, do you have a recommendation on an efficient way to find & correct bad ticks in the data before running the script on the large CSV file?

    • #418 written by birt March 5, 2014 (6 months ago)

      There are gaps in the data and there’s not much you can do about it.

      Regarding the CreateXX parameters, they’re only for the FXT files for those timeframes. The HST files will be created whether you enable them or not.

      Correcting bad ticks is not necessary. The CSV2FXT script will filter out such ticks automatically (e.g. if the price jumps more than 10% from one tick to another).

      • #419 written by jivetrader March 5, 2014 (6 months ago)

        still confused. so if i intend to backtest on say M15, M30 and H1 timeframes, i should select True for these 3 timeframes and i am all good, or do i still need to run the script 3x ?

        >>Regarding the CreateXX parameters, they’re only for the FXT files for those timeframes. The HST files will be created whether you enable them or not.<<

  • #420 written by birt March 5, 2014 (6 months ago)


    jivetrader:

    still confused. so if i intend to backtest on say M15, M30 and H1 timeframes, i should select True for these 3 timeframes and i am all good, or do i still need to run the script 3x ?

    You’re all good, you don’t need to run the script 3x.

    • #421 written by jivetrader March 5, 2014 (6 months ago)

      thanks so much for the hand-holding Birt!

      when moving the HST files from my ‘connected’ MT4 to a new install of the program (for offline backtesting only), there are no subfolders for brokers yet in \history

      can i just create a folder of any name?

      • #422 written by birt March 5, 2014 (6 months ago)

        I haven’t tried that but I suspect it won’t work. I would suggest connecting once and then deleting the account. MT4 will use the folder of the broker that it was last connected to.

  • #423 written by Vinisius March 5, 2014 (6 months ago)

    But if we have an Ea that need to verify real account then as we do this ?.

    • #424 written by birt March 5, 2014 (6 months ago)

      You can’t backtest in offline mode if the EA needs to verify the account.

  • #425 written by Vinisius March 5, 2014 (6 months ago)

    Then if I perform backtest online, I must first mark as read only manually HST files … but then when I open a chart of eurusd in real account, for example 1M TF, this will generate an error in MT4 ?

    • #426 written by birt March 6, 2014 (6 months ago)

      You only need to mark the HST files read-only if you’re backtesting recent periods. If your backtests are using the M1 timeframe and you’re not testing the past 1-2 days, you don’t need to worry. Furthermore, as best as I can tell if you don’t have any charts open, no data gets overwritten.

      As for MT4 with the HST files marked read-only, it appears to work fine at the first glance but I suppose it’ll only work fine for as long as it has room in its memory buffers. I wouldn’t recommend trading live (or even forward testing) with the HST files set read-only. Simply make a copy of your MT4 installation and login with the investor password.

  • #427 written by Vinisius March 6, 2014 (6 months ago)

    OK.

    but then it is not necessary to put account´s number … , only the investor password login ?.

    • #428 written by birt March 6, 2014 (6 months ago)

      If you try logging in with the investor password, you will notice that you still need the account number.

  • #429 written by Jelks March 10, 2014 (6 months ago)

    Hi Birt,

    This time of year makes me cuckoo. It all has to do with US Daylight Savings Time.

    We’re on FXCM in New York City. I do get it that many brokers, including FXCM, want to keep 5:00 Eastern time US synced with 00:00 server time.

    So, Last week, 5:00 PM Eastern *Standard* Time was 00:00 (midnight) server time, 7 hours later. FXCM’s MT4 was on *GMT+2*.

    Today, Sunday evening, just after Daylight Savings Time, 5:00 PM Eastern *Daylight* Time is 00:00 (midnight) server time, also 7 hours later. FXCM’s MT4 is now on *GMT+3*.

    So, to keep that 5:00 PM Eastern time (Standard *or* Daylight) synced with midnight (server time), when Daylight Savings Time happens as it did today, they have to change the GMT offset.

    Our EAs are based on particular hours of the day. So when we download a bunch of CSV files for five or six pairs, we backtest *each* of 24 hours for each pair using our default inputs. We then optimize the most promising looking of those hours. Those could be from one to six hours (generally) out of each pair. The initial backtesting with default inputs takes a couple of hours (considering the updating of our spreadsheets); the optimization of each chosen hour can take up to a almost a whole day! So you can see it’s a very time-consuming process.

    A week ago, we downloaded the most recent tick data through March 1st. After having run CSV2FXt on our pairs — with FXTGMTOffset set to 2 — we backtested and optimized. I did about half the pairs we wanted to, then Daylight Savings Time happened today! Uh oh, we’re now on GMT+3!

    So does this mean we now have to re-run CSV2FXT for all the CSV files, changing FXTGMTOffset from 2 to 3, and then backtest and optimize with the “new” hours from the beginning?

    Jelks

    • #430 written by birt March 10, 2014 (6 months ago)

      If your broker is at GMT+2 during the winter and the time switch occurs according to the US rules, you should set FXTDST to 1 and FXTGMT to 2. This will make your FXT have GMT+2 during the winter and will switch to GMT+3 during the summer.

      • #431 written by Jon March 10, 2014 (6 months ago)

        So here’s the problem. Doex FXT file store when the GMT will offset for the broker? Because I’ve seen sometimes where it will still have extra bars on saturday. With this strategy you are suggesting above, it should be GMT +8

        So is the FxTDST update correctly mirroring the new USA DST standard? How would a user test this?

        • #432 written by birt March 10, 2014 (6 months ago)

          The data in the FXT is pre-shifted with the DST. The conversion script has the rules built into it.

          It would not be GMT +8, it would be GMT +2 with US DST (so GMT+3 during the summer).

      • #433 written by Jelks March 11, 2014 (6 months ago)

        Ok, thank you Birt. I should have noticed this much earlier. :)

        Jelks

        • #434 written by Jelks March 11, 2014 (6 months ago)

          But I have to ask .. if FXTDST was formerly set to 0, and the “best” hours reported were (say) 6 and 12, does that mean that those hours are now off by 1 due to DST?

          I have updated my FXCM CSV2FXT settings file so this (hopefully) shouldn’t reoccur in the future, but it would be nice to know when I’m testing that 6:00 in the winter means the same as 6:00 in the summer, regardless of whether DST is in effect or not. Does setting FXTDST to 1 accomplish this?

          I guess a more apropos question is: if I have a settings file on the server called (say) eurusd 06.set — a settings file that was optimized using FXTDST set to 0, where the chosen hour to place the trade is 06:00 — should that hour remain the same, or increased by 1 now that DST is in effect?

          I realize I am am going to have to backtest all hours with defaults then optimize the best hours on the new FXTDST setting, but I am just trying to get a clear idea of what’s going on before I go on that venturesome journey. Could I/should I, for example, open up that eurusd 06.set file and now change the “ChartHourToTrade” input to 7?

          Thanks,

          Jelks

          • #435 written by birt March 11, 2014 (6 months ago)

            If it was optimized using an FXT with no DST, your EA was trading at 7 (US time) instead of 6 during the summer. I can’t tell what that means for your EA.

            I wouldn’t change the set without further backtesting. The question you have to ask is not necessarily “which hours are statistically better for my EA?”. I would rather ask “when do I want my EA to trade?” – is it between the US and the Asian session? If so, you’d want it to start trading sometime after 5PM New York time and stop trading before 7AM or 8AM Tokyo time.

            What I’m saying is that if your EA is designed to trade between sessions, it won’t make a huge difference if you start it at 6 or at 7 as long as the starting time is outside the US session. Of course, starting it earlier will probably result in more trades. So, my advice would be to leave the set file as it is for now and start those backtests.

            Finally, you could take a look at http://fxtrade.oanda.com/analysis/market-hours or any other market hours widget thing.

  • #436 written by Dean March 15, 2014 (6 months ago)

    Hi Birt

    I have been updating TDS on a new computer and have run into a series of problems. I am using mt4 build 610 (which is new to me) and I have downloaded and installed the TDS 1.3.1 and the csv binaries v46.

    I have put the csv files into the corresponding places in mt4. You usually name the csv files as the places they need to go such as MQL4/include/FXTheader.mqh with the file to be copied being FXTheader.mqh. This has been placed in mt4/MQL4/include/. I have copied the other parts of the csv v46 in the same way. Scripts to scripts and libraries to Libraries.

    Once I load my mt4 with TDS (or on it’s own) I do have CSVtoFXT anywhere in my mt4 navigator. Usually on an older mt4 builds it appears here and I can the use it to create my fxt’s.

    I must be doing something wrong ?

    Also I am trying to workout where to put the dukascopy csv file. You say to put it into MQL4/files but my mt4/MQL4/ has no files folder. it has MQL4/experts MQL4/include MQL4/indicators MQL4/libaries MQL4/Scripts and that is it. There is a files folder in MQL4/include/files I am wondering if this is where the csv file should be placed.

    Many thanks

    Dean

    • #437 written by birt March 15, 2014 (6 months ago)

      You’re using the wrong MQL4 folder. The new MT4 builds use a different path, go to File -> Open Data Folder to find out where.

      Alternatively, look into the /portable switch.

  • #438 written by Dean March 15, 2014 (6 months ago)

    OOPs paragraph 3 should say (or on it’s own) I do NOT have CSVtoFXT anywhere.

    question should make more sense now?

  • #439 written by Dean March 15, 2014 (6 months ago)

    Thanks birt

    Went into file open data folder now works a treat.

  • #440 written by Jon April 14, 2014 (5 months ago)

    For the spread padding: is it possible to add a parameter to specify a minimum spread? Often times [market making] brokers who include the commission in the spread, and the spread is still variable often have a minimum spread that they offer to their clients. So this makes spread padding by itself difficult to tune precisely.

    I hope I explained this correctly.

    • #441 written by birt April 14, 2014 (5 months ago)

      Good idea. This feature has been added to CSV2FXT v0.48 which will be released when more significant changes make their way in.

  • #442 written by Jon April 14, 2014 (5 months ago)

    I also noticed another bug. When you select a custom start date and/or end date, Instead of converting the data from the start date, it will still read the entire file. For example, I might set the start date to 2010, but I will see “2014.04.14 10:55:50.738 CSV2FXT EURGBP.,M5: Starting to process EURGBP. 2007… ” In the expert log files. Why would you need to process 2007 if I only need data from 2010 afterwards?

    Maybe it could be a bug in the experts/alerts logging, but I doubt it. I want to be sure there is no unnecessary time processing unneeded data.

    Sometimes I need to split data into several files. If it has to read the entire csv file each time, this might be slowing things down.

    • #443 written by birt April 14, 2014 (5 months ago)

      That’s not a bug. The CSV file is read sequentially and thus the script has to go through the file record by record until it reaches the start date of your choice. It would be possible to develop a heuristic algorithm for arbitrary file seeking within the CSV but it’s beyond the scope of the project.

      One thing to note is that if you enable Create HST files, these will contain ALL the data in the CSV regardless of the specified start date (which only applies to the FXT). Disabling this parameter might result in a small speed increase when using a custom start date.

  • #444 written by Jon April 15, 2014 (5 months ago)

    Unfortunately, disabling create HST files doesn’t seem to speed anything up :(

    Glad to see you updated to handle spread padding with a minimum spread. Many brokers will still charge a minimum regardless of what the ‘real’ spread is.

    • #445 written by birt April 15, 2014 (5 months ago)

      The speed increase associated with that is indeed minimal because the HST files are quite small.

  • #446 written by Vinisius April 22, 2014 (5 months ago)

    CSV2FXT v0.48 already here. Now spread padding must be configured in a different way?.

    Birt, can explain you how to do this?

    Thanks.

    • #447 written by birt April 22, 2014 (5 months ago)

      Spread padding works just like before.

  • #448 written by Vinisius April 22, 2014 (5 months ago)

    Sorry for my English.

    My broker Activtrades has minimun variable spread of 1.3 pip. How do I configure the spreadpadding to take account of the minimum spread ?.

    Thanks.

    • #449 written by birt April 22, 2014 (5 months ago)

      If you’re using real spread, you could specify 1.3 for the minimum spread parameter.

  • #450 written by JiveTrader April 22, 2014 (5 months ago)

    i used CSV2FXT 0.46 to create my files. how do i know if i need to create new ones using 0.48?

    • #451 written by birt April 22, 2014 (5 months ago)

      If you care about the occasional order having an error of 0.1 pips, you should recreate your FXT files. The effect will be most visible on scalpers and other EAs that only trade very few pips at a time.

  • #452 written by Vinisius April 22, 2014 (5 months ago)

    OK.

    Sorry for my english.

    But this minimum spread (1.3) shall be equal to the minimum spread of Dukascopy and then from here my minimum spread will be variable whenever the minimum spread of Dukascopy also moves up from its minimum?

    If this is not so, Dukascopy has minor variable spreads to my broker and then either the variable spread will not move 1.3 and then the will be fixed and non-variable spread.

  • #453 written by Carmine May 1, 2014 (4 months ago)

    I have exported in MT4 the tick datas and of course i have selected to generate HST files also! It is normal that the FXT files generated are much bigger than the corresponding HST files (for example the fxt is 7GB and the HST is 140 MB)!

    • #454 written by birt May 2, 2014 (4 months ago)

      Yes, that’s perfectly normal. The HST files only contain bar open, high, low and close values so the largest HST file would be the M1 history with 4 prices per minute. FXT files have 4 prices for each tick…

  • #455 written by Carmine May 2, 2014 (4 months ago)

    Birt, many thanks for your prompt reply! Since i plan to refresh periodically my FXT files to be updated to the most recent datas, of course i must first delete the FXT files in my appropriate MT4 tester folder and then export again the FXT files (by your scripts or Tickstory) again in the folder! I must also delete the HST files and refresh them? If not when exporting the FXT files, i must alwasy select create HST files, or this can create conflict?

    • #456 written by birt May 2, 2014 (4 months ago)

      You should always create the HST files when recreating the FXT. The new HST files should overwrite the old HST files.

  • #457 written by Carmine May 4, 2014 (4 months ago)

    I was trying to prepare the csv files with your CSV2FXT script (since i am usinfg a 625 build, i am usinb the newer version f the script 0.48); the CS file is in the correct folder (mql4/files), but when click ok on the panel of the scrip a receive a warning “comma issues detected” and after “bad csv format. aborting. The CSV was created with SQ tick data downloader and it has a csv extensio (GBPSD.csv)
    hat could be wrong?

    • #458 written by birt May 4, 2014 (4 months ago)

      Most likely the export format is wrong. It’s hard to tell without seeing a couple of rows from the CSV file.

  • #459 written by Carmine May 4, 2014 (4 months ago)

    I have exported again the CSV file and now everything is fine! Maybe the error was due to the faxct that i have copied 2 CSV files (EURUSD and GBPUSD) in the files folder at the same time or to the fact that i forget to delete from the tester folder the old FXT file! nyway now everything is fine! I have decided to use the CSV2FXT script instead of Tickstory to create the FX because i am having on my eas results too good and i want to check if using your script i will have the same results (of course i will select the same spread values)! Thanks

  • #460 written by Vinisius May 15, 2014 (4 months ago)

    Hello Birt.

    I as I told you in an email there is a bug in Minimum spread option.

    Does not work properly. If I put for example 0.3 appears spread 3000, if I put 0.4 spread 4000 etc…

    If you’re going to fix this bug out a new version of the SCV2FXT script might take advantage and add a new option to remove maximum lots permitted by the broker??

    Thanks.

    • #461 written by birt May 27, 2014 (3 months ago)

      This bug was fixed in CSV2FXT 0.49, which was released the same day as you posted but I forgot to post about that here. If you didn’t get the update already, you should.

  • #462 written by Vinisius May 15, 2014 (4 months ago)

    Not only happens to me.

    Other person, friend of me too happens it.

    Thanks.

  • #463 written by Jon May 27, 2014 (3 months ago)

    1) I wanted to know if FXT files can be used across brokers? In other words, can I use a Excel Markets-generated fxt file in XeMarkets-tester. Anything to look out for?

    I want to avoid having to create too many separate fxt files, especially over multiple years of data.

    2) if I use a fixed spread broker and I already have an fxt file with variable (real) spreads. do I need to create a separate fxt file to test fixed spreads?

    • #464 written by birt May 27, 2014 (3 months ago)

      1. Yes but you have to keep in mind that the FXT will inherit some values from the broker used at its creation (swaps, min stop distance etc.).

      2. You have to create a separate FXT for backtesting with fixed spread.

  • #465 written by Abhay May 28, 2014 (3 months ago)

    How do I get the updated version for $28 to run on MT4 646.?

    • #466 written by birt May 29, 2014 (3 months ago)

      I sent you an email with the details.

  • #467 written by Luca June 3, 2014 (3 months ago)

    Hi Birt.
    when I run the script on a small portion of the file, it still takes long time, and also seems to be checking periods outside of the time I specify ( it warn me about gap before the specified period … )
    Is it normal ? Thanks.

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top