Table of contents

Converting to FXT


1. When running CSV2FXT, I get the message “Could not open input file”. What should I do?

That can only be caused by the script trying to open a file that is not there so you either copied the CSV to the wrong folder, named it incorrectly or typed its name incorrectly. Check the following things:

  • You are using the correct CSV2FXT version. v0.43 and below are only suited for MT4 builds 509 and below. You must use a newer CSV2FXT with MT4 builds 545 or higher.
  • You put the CSV file in the correct folder (experts\files for MT4 build 509 or lower respectively MQL4\Files for MT4 builds 545+).
  • If using MT4 build 509 or earlier, you put the CSV file in the correct MT4 installation folder (if you have more than one).
  • If using MT4 build 545+, you put the CSV file in the correct MT4 data folder (head to File -> Open Data Folder to find out where it is).
  • If you left the CsvFile field blank, check that the file is named as your pair and has a csv extension.
  • If you typed something in the CsvFile field, make sure that you typed the full file name including its extension.
  • If your explorer is configured to hide the file extension for known files, your file might be named EURUSD.CSV.CSV and explorer might just show EURUSD.CSV, or your file might be correctly named EURUSD.CSV and your explorer might just show EURUSD. In Windows 7, to work around this issue you can hit ALT in an explorer window, click Tools -> Folder options -> View then uncheck “Hide extensions for known file types”. The procedure is similar for other Windows versions.
  • If you are using Vista with MT4 build 509 or earlier and your MT4 installation is in Program Files, I suggest making a copy somewhere else (for instance on your Desktop) and working from that one instead. Vista has something called UAC that can be a royal pain in the ass. You can also disable UAC if you wish, read a few paragraphs below on how to do that.


2. How can I change the fixed spread in an existing FXT file?

Use a hex editor (e.g. WinHex, Neo Hex Editor) and change it at offset 0xFC. Note that it’s going to be displayed in hexadecimal so you’ll probably have to use a calculator.


3. How can I change [some other setting] in an existing FXT file?

Look up the offset in FXTHeader.mqh, it’s displayed next to each setting. Use a hex editor. Note that all values are in little endian format, meaning that if you have a value that exceeds one byte you have to fill in the bytes to its right. For instance, if you want to write 300 in the file you will have to write it as 2C 01.


4. I cannot find the resulting FXT and HST files. Where are they?!

For MT4 builds 509 or earlier, this is typical for Windows 7 / Windows Vista users that run MT4 from their Program Files folder. The issue is caused by the fact that UAC is enabled and these operating systems use folder virtualization. For Windows 7, the resulting files can be found in c:\ProgramData\ typically, while for Windows Vista they can be found in c:\Users\username\AppData\Local\VirtualStore\Program Files\. If they’re not there, just search for *.FXT, they must be somewhere. To get around this problem, either copy your MT4 folder in a location that’s not protected by UAC (e.g. your desktop) or simply disable UAC (type UAC in the start->search box thing and follow the on-screen process).

For MT4 build 545 or later, head to File -> Open Data Folder and look in MQL4\Files.


5. Backtesting an EA with my newly created FXT results in Order Send Error 131. Why?

This might be caused by the lack of a MIN_LOT and LOT_STEP for the symbol in the resulted FXT file. This happens when generating the FXT file with a MT4 client that wasn’t connected to the broker when the script was started. As specified above, your terminal should be connected to your broker when generating an FXT file.


6. What do I do if I get an alert saying Bad CSV Format when I try to convert the CSV I exported from JForex?

This is in all likelihood because you either failed to select a delimiter or selected a dot (.) as the delimiter, so the solution is easy: you go back to JForex, select comma (,) as the delimiter and export the CSV once more. This should be much faster than the first time since the data is cached. If you’re uncertain about the other parameters, take another look at the JForex downloading guide.


7. Is it possible to append tick data to an existing FXT file (and the accompanying HST files)?

No.


8. Is it possible to append tick data to an existing CSV file?

Yes. The CSV2FXT script will disregard duplicate information and even the JForex header line if it is encountered in the middle of the CSV file.

If you are using JForex, you should start your CSV from the time your older CSV ends. You can delete the header row from the new CSV file before concatenating or you can simply skip this step, in which case the CSV2FXT script will skip that line and warn you about it in the experts log. When the export is finished, simply append the new CSV to the older one.

If you are using the PHP scripts, it is ideal to stop your processing at the end of a month. The processing script automatically appends to an existing file so if you resume from the start of the next month your CSV will be perfect. However, you can stop your processing even during the month; for example, if you have a file named EURUSD.csv that ends on 19.03.2012 and then append data starting from 01.03.2012 and ending with 02.04.2012 (e.g. by typing php process_dukascopy_data.php EURUSD 201203 201205 EURUSD.csv), the new data will be appended to your existing CSV and the data between 01.03 and 19.03 will be in your CSV twice. In such a case, the CSV2FXT script will output about 20 errors in your log and spit out an alert about older ticks, but it is smart enough to skip the duplicated data so the resulting FXT should be perfectly fine (although it will take a little bit longer because it has to skip the duplicated period).


9. Why does the CSV2FXT script get stuck as soon as it is started? The progress indicator is not displayed on the chart and the only message displayed in the experts log is “The date column appears to be 0. Sample…”

Most likely this is because you exported the CSV file using JForex but failed to select the comma (,) as the field delimiter as mentioned in the guide. Since for some reason that I fail to grasp the default JForex setting is to use a space as field separator and put several commas at the end of each line, your CSV file is useless and has to be regenerated. The good news is that JForex caches the data so you don’t have to wait for the download once more.


10. How could I change the timeframe of an existing FXT file?

Two steps:

  1. Since the file name contains the timeframe, for instance EURUSD15_0.FXT for an M15 EURUSD FXT file, you have to rename (or copy) the file for the desired timeframe. Assuming you’d want to make it H4, you translate that to minutes (4 hours = 240 minutes) and you rename the file to EURUSD240_0.FXT
  2. Next you have to grab a hex editor and edit the file at offset 0xD0. Change the number to the same one used in the file name in the previous step (but make sure you convert it to hex; using our example above, 240 is 0xF0 so you would have to change it from 0x0F – which is 15 – to 0xF0).

One thing to note that if you perform this change is that the bar count displayed in the results is going to be wrong and indicators running on the backtest timeframe will be misleading because the ticks are stored as part of bars and the bars will be for a different timeframe. Essentially, if your EA has any indicators running on the backtest timeframe, this method is not safe to use.


11. Can I create range bars or Renko bars with CSV2FXT?

Not with the version I created. However, Artur from AZ-Invest has developed a version of the script that works with Renko bars.

Using the tick data


1. Why are my FXT files being overwritten?

You probably didn’t start the MT4 terminal via the Tick Data Suite and your FXT files are not read-only for some reason.


2. Why is my terminal creating an FXT file when I start backtesting?

  • Did you create the FXT file for the currency pair & timeframe that you are trying to use in your backtest?
  • If the answer to the above is yes, did you copy it to the correct location (in recent MT4 builds, head to File -> Open Data Folder to find the right place)?
  • Also see the answer for question #1 above.


3. My backtest runs but I do not get 99% modeling quality. Why?

This is in fact the same question as #2 above.


4. My backtest stops as soon as I click the start button. The backtesting journal tab displays TestGenerator: internal error because the file not opened. What is wrong?

That’s MT4’s way of notifying you that it refuses to overwrite an FXT file that is set read-only and it happens with MT4 builds 509 or earlier. You need to close the MT4 terminal and start it via the Tick Data Suite shortcut.


5. Why does my terminal crash as soon as I press “start” in the backtest user interface?

If you’re not using the latest MT4 build, it’s most likely because the EA was compiled with a build newer than yours and Metaquotes did not think it wise to add ex4 versioning. You can tell this is the problem if attaching the same EA to a chart will also result in a crash. The solution is to use the latest build or get the EA compiled with your build.


6. Why do I get a red bar at the end of the backtest even though my modeling quality is 99%?

Because the FXT you are using was created with an older script. It’s nothing to worry about, though – you can safely ignore it.


7. What can I do if my optimization results window is empty even though MT4 appears to run multiple passes correctly?

That means the results you obtained were not profitable. If you wish to see them nonetheless, you can select the Optimization Results tab, right click inside it and deselect “Skip Useless Results”.


8. I am getting empty results (no trades) in all my optimization passes when using the Walk Forward Analyzer. Is there a workaround?

This only happens with some EAs that need the Metatrader 4 terminal to be connected to the broker even when backtesting, for instance Wallstreet Forex Robot or FAP Turbo. To put it simply, since the FXT is already there, the backtest starts too fast. The only solution for this problem is to add an artificial delay, which can be done by using the Configuration program installed with the Tick Data Suite. Just fire up Configuration and increase the Backtest delay factor to something like 3. This will start the backtest with a sensible delay which should be enough for the terminal to connect to the broker. In case it isn’t, feel free to increase it to something like 10. Set it back to 0 once you’re done with that particular EA unless you want to have a permanent delay when backtesting or optimizing with tick data. Note that this option will not have an effect on Metatrader 4 terminals that are already running and you will need to restart these to take advantage of it.


9. Why are all the volumes in my tick data backtest smaller than 1?

It’s because you are using real spread when creating your FXT with MT4 build 509 or earlier. When enabling this option, the spread is stored in the volume field. If your strategy makes use of the volume number (which is simply the number of ticks in MT4), you need to use fixed spread or employ other strategies such as counting the number of ticks and storing it in an array.


10. Why are all the volumes in my tick data backtest huge numbers?

Same reason as for the previous question, but it only applies to MT4 builds 545 or higher. To get the actual volume, simply apply a bitwise AND 0xFFFFFFFF like this:


volume = Volume[0] & 0xFFFFFFFF;


11. Why does my backtest fail as soon as I attempt to start it with the backtest journal saying “No data for testing.”?

This typically happens when you enabled Use date and the start and end dates you selected are outside the range covered by your FXT.


12. My backtest stops somewhere in the middle, even though the FXT file I created spans a longer time period. How can I fix this?

This typically happens when you’re using MT4 build 600 or higher, you created an FXT file larger than 4GB and you forgot to run MT4 via the Tick Data Suite (the MT4 window title bar should say “… with Tick Data Suite vX.XX”).

If this happens and none of the above are the causes, you should first attempt to backtest the MACD EA using your FXT and if it stops at the same point, you should investigate the source CSV in that range and check the log of the CSV2FXT script for any potential errors.


13. How come my backtest results differ from an optimization cycle results if I am using the same parameters?

This only happens when you are using a start/end date for your optimization and you’re using MT4 build 509 or earlier. Sadly, this is a bug in Metatrader 4: only the first pass of an optimization uses the start and end date you selected; subsequent runs use the full range available in the FXT. This bug was fixed starting with MT4 build 545.

In any case, with MT4 builds 509 and earlier, to get the same outcome when backtesting with the parameters for one of the optimization results, you can either make an FXT that spans exactly the period that you want to optimize for, or alternatively you can simply disable Use date when running the backtest for the selected optimization results (if it’s not the parameters for the first run, that is).

  • #217 written by Kirill April 22, 2014 (2 years ago)

    Hi Birt,

    No worries, glad I could help.

    Question in regards to your previous comment:

    “If you want to have two different FXTs for the same timeframe in the same MT4 installation, I would recommend looking at the /portable switch.”

    I do not understand how the /portable switch will help with two different FXTs for the same timeframe. Could you please elaborate a little on this?

    Thanks,
    Kirill

    • #218 written by birt April 22, 2014 (2 years ago)

      You can make two different shortcuts, one with /portable and one without /portable and you can store the first FXT in the installation folder and the other in the AppData folder.

  • #219 written by Elio Matacena April 26, 2014 (2 years ago)

    Hi Birt,

    good evening. I need to understand one line that comes out when the BT starts:
    “Tester: exchange rate cannot be calculated”.

    Could you, please, explain me what it means and what are the effects on the BT results?

    Best regards,
    Elio Matacena.

    • #220 written by birt April 26, 2014 (2 years ago)

      That typically happens when you disabled one of the pairs involved in the cross calculation. For instance, if you’re backtesting USDCAD and the currency configured in the backtest is JPY but you don’t have the USDJPY pair, you will get that error. I’m not sure what the effects are on the backtest, haven’t tried to find that out because it’s an MT4 issue, not a TDS issue.

  • #221 written by Long J. Silver June 18, 2014 (2 years ago)

    Hi Birt…I have been a user of TDS for a whole year now and currently I am using TDS 1.3.2 for optimization purposes. It’s only in this latest version that I have come across a problem. Let me explain it. I am optimizing four parameters P1, P2, P3 and P4 and for each there is a starting numerical value, a step value and a stop value. When I run TDS, I find the best results 200 for {P1=4, P2=8, P3=12, P4=16}. Now I simply change the starting value of each parameter and relaunch the TDS. This time again, I find the best results for {P1=4, P2=8, P3=12, P4=16} but the result = 165. This is very strange because the result should be 200. For the same paramters, when I run TDS in non-optimization mode, I get yet another result which is 175. What I want to make clear is that for all three runs, the four parameters and all other parameters are exactly the same. I have had this situation in all the previous versions but the result was always the same whether or not I launch it in the optimization mode. I hope I am clear in my explanation. The presence of discrepancy gets confirmed because sometimes, I see another problem too. When I get an optimized set of parameters, I just decrease/increase the starting/stopping value to increase my sample space and I do not catch the previous optimized set of parameters at all as if TDS is skipping it. In all experiments, I am using a start/end date but the FXT file contains data only for that span of time. MT4 build is 646. Could you please look into the issue. I shall be most grateful. I remain at your disposal for any further info. Thanks and regards, Long J. Silver

    • #222 written by birt June 18, 2014 (2 years ago)

      Are you perchance using genetic optimization? Or perhaps did you enable slippage during optimization in the TDS configuration? Other than slippage, there’s no other reason for TDS to have an impact on optimization results.

      I would recommend attempting to reproduce the problem with a single parameter instead of 4, saving the results and ensuring that the result that gets selected the second time is among the results that were displayed the first time. I would also advise running a regular backtest with each set of parameters. Finally, I’d also recommend running a few optimizations without TDS to see if the issue persists.

      If I perchance misunderstood the issue you’re experiencing, I kindly ask you to send an email to the support address attaching screenshots and all the saved optimization results involved in the problem.

      • #223 written by Long J. Silver June 18, 2014 (2 years ago)

        Thanks Birt for your comments….I think the solution lies in the first paragraph of what you wrote.
        Slippage is not enabled but the ‘genetic algorithm’ box is checked in MT4 so it’s enabled…I have no idea what this genetic algorithm is all about. Could you please shed some light on it…Thanks a million…L. J. Silver

        • #224 written by birt June 19, 2014 (2 years ago)

          Please take a look at the link posted by achab below.

          Long story short, genetic optimization will not go through all the steps selected, it will only select some of the optimization steps and the selection is more or less random so it’s normal to get different results in different runs.

          • #225 written by Long J. Silver June 19, 2014 (2 years ago)

            Thanks a million Achab and Birt..I now understand why some parameter steps were skipped…However, having different reults for the same parameter set is still not clear. I am going to deactivate genetic algorithm to get rid of any random effects et redo the tests. Anyhow, a related question: Is there any limit on the number of combination of parameters which can be used for optimization cycle? In the article, a very high number of combinations is mentioned. I start to have problems even at 1280 combinations, sometimes it leads to a restart of my computer. Thanks and regards, L. J. Silver

            • #226 written by birt June 19, 2014 (2 years ago)

              Yes, there is a limit but it’s much higher than 1280. I can’t seem to remember it now, though.

              Anyway, if you get crashes or restarts it’s likely an MT4 issue that was introduced with the new builds (600+). Some other people have had success in solving these problems by underclocking their CPU and/or RAM. Another thing you could try is to compile your EA with one of the old builds (e.g. 509) which forces MT4 to use the old backtesting engine.

              • #227 written by achab June 20, 2014 (2 years ago)

                Hi Long J. Silver,

                good morning. I have a question for you: can you list us your hardware and O.S. complete details, please?

                Thanks in advance.

                Best regards.
                /achab

                • #228 written by Long J. Silver June 21, 2014 (2 years ago)

                  Hi Achab and Birt. Thanks Achab for your interest. First of all the system specs: Windows Server 2008 R2 Standard 64 bit (Service Pack 1), AMD Opteron Dual Core 2.3 GHZ, RAM 3 GB. Please tell if any further info is needed.
                  The results obtained for the same set of parameters are different with and without the activation of the genetic algorithm. I am tempted to go for the results without genetic algorithm.

                  • #229 written by birt June 21, 2014 (2 years ago)

                    Let me clarify.

                    Genetic algorithm enabled means that some passes are skipped and consequently you might not find the absolute best result because that pass might be skipped.

                    Genetic algorithm disabled means that all passes are processed and your best result will indeed be the best result of all the passes.

                    In any case, it really has nothing to do with the Tick Data Suite.

                  • #230 written by achab June 21, 2014 (2 years ago)

                    Hello,

                    and good evening. Be advised that such cpu is not so performing, I suggest you to use a desktop, not a VPS for BT.

                    Bye.
                    /achab

  • #231 written by achab June 19, 2014 (2 years ago)
  • #232 written by Long J. Silver August 3, 2014 (2 years ago)

    Hi Birt
    I prepare Tick data by first downloading the .csv file, then running CVS2FXT by giving filename.csv, choosing real spread and then choosing H1 while keeping all options as they are.
    I carried out proper trading on a demo account and I have no losing trade for the first six months of 2014 making around 800 pips. However, when I backtest for the same period, I make only 230 pips..A number of negative trades are there in the backtest which were not there on my trading using a demo account.
    When I investigate, I see that there are slight price differences (single pip or lesser difference) which are causing the difference between the trading results.
    Do you know any method to remove this discrepancy? May be I should not choose real spread while preparing tick data. However, when I do that today (weekend), it asks me whether I would like to use the weekend spread of 14.5 pips.
    Guidance requested…I shall be most grateful..
    L. J. Silver

    • #233 written by birt August 3, 2014 (2 years ago)

      You cannot have 100% identical results unless you are using the tick data from your live broker and there was no slippage at all for your trades. This becomes very obvious especially for EAs that are very sensitive to sub-pip differences and I assume your EA falls into that category.

      You could start gathering live tick data from your broker which would solve the issue to an extent. If you wish to use fixed spread and you’re doing this over the weekend, you should probably set the spread manually (the Spread parameter) to the desired value.

      Another option would be to use a different source of tick data that better matches the price feed of your broker, you could try the one from Integral for example.

      Finally, you will still have small differences even if you use your broker’s data due to slippage. There’s no way to make it match 100% due to this variable; the best you can do is configure slippage to match your broker’s and get an estimative result.

  • #234 written by Jon September 16, 2014 (2 years ago)

    I was testing TDS on a different laptop today and it worked perfectly on one mt4 folder with 2 years worth of tick data already prepared (csv2fxt 0.49 build 670 or 646) in Excel Markets broker from a different laptop and drive).

    Then I was using a different set of data (prepared the same way, just a different mt4 folder, same broker Excel). I launched with TDS and something weird happened:
    TestGenerator: unmatched data error (high value 1.61238 at 2014.09.09 11:30 is not reached from the least timeframe, high price 1.61107 mismatches) and the backtest would run, but finish quickly and only run on a few days of the entire 2 years.

    So I restarted the terminal and then got this message:
    Tester: exchange rate cannot be calculated then shortly afterwards
    2010.01.04 00:00 zero divide in '!ea.mq4' (6768,48)
    2010.01.04 00:00 Testing pass stopped due to a critical error in the EA
    GBPUSD,M5: 3 tick events (149759 bars, 28465159 bar states) processed within 187 ms (total time 234 ms)

    This never happened before ever, so I actually reinstalled mt4 on that specific mt4 folder. Before it was just a copied folder, so maybe that would fix things.

    Then something weirder happened on the next run. it started to download 1m, data and generate its own fxt file prior to the first test. I figured that was impossible, but I looked closely and saw that the usual filename ‘GBPUSD.5_0.fxt’ was now ‘GBPUSD5_0.fxt’ . I checked in the log history and found that each run on this rogue terminal was
    actual tick file "I:\Excel 4 Tick 2010_2012\tester\history\GBPUSD5_0.fxt" found
    I tested with a different pair that I already had history for and verified that it did generate a new fxt file without the period ‘.’

    The EA then would run with the new fxt, but not exactly what I wanted. So I then decide to make a copy of the old fxt file and rename it GBPUSD5_0.fxt . I then set the file to read-only and restart the test. Now suddenly it works, but I still get the exchange rate cannot be calculated error.

    Why did mt4 just ignore the standard .fxt filename structure in this terminal? (the other terminal, same computer,same broker backtest ran the first time, no issues) That was an hour of my life I cannot get back.

    ——-
    Does running from a USB drive make a difference in performance?
    ———
    BTW, it would be nice if users could preview comments on this board before submission.

  • #235 written by Jon September 16, 2014 (2 years ago)

    I had all pairs listed in the market watch, and restarted the terminal. Still get the error ‘exchange rate cannot be calculated’ My ea still runs fine, and I have a method to define pip value within EA parameter if autodetect is an incorrect value.

    • #236 written by birt September 16, 2014 (2 years ago)

      The currency pair name gets stored in the FXT file. I can only assume that the problem stems from the FXT being created with a currency pair that lacks the dot suffix. When using suffixed currency pairs, it’s always a good idea to recreate the FXT using exactly the currency pair you are targeting. Generally the “exchange rate cannot be calculated” error only appears when MT4 is unable to convert from your currency pair to the base currency configured in the backtest but if you selected a base currency that had the required conversion symbol in the market watch window, it must be something related to the suffix (or lack thereof).

      I’m not entirely sure why things worked as they did, but generally MT4 looks for the symbol listed in the backtesting engine, not for the symbol listed in the market watch window.

      • #237 written by Jon September 16, 2014 (2 years ago)

        I think I figured it out. In the laptop in question, I saw that if I did not try to pick a specific date range for the backtest (leave ‘use date’ unchecked), the backtest would run. It was only if I tried to pick a specific date range OR [sometimes] do the visual mode on. I then concluded that it was the laptop has a different Windows regional date/time settings. It was set to Thai [thailand], which uses Buddist calender. So I changed it to english (US) which is what the computer that generated the fxt files was under and restarted the terminal. I was then able to use date ranges just fine.

        The fxt filename missing the period ‘.’ in one of the mt4 installations. That is new. Same broker and everything. But you can change the filename….just make sure read-only is set.

        The Terminals were identical (same broker and mt4 build # that generated fxt files were used).

        Never a dull moment with mt4 sometimes.

  • #238 written by Kirill September 17, 2014 (2 years ago)

    Hi Birt,

    I haven’t been using TDS for a couple of months now (not enough time atm). I was wondering if there is a way to freeze my subscription.

    How do I pause the monthly payments and then renew them when I start using TDS once again?

    Thanks,

    Kirill

    • #239 written by birt September 17, 2014 (2 years ago)

      Certainly. I sent you detailed instructions for the cancellation via email. Reactivating it is possible with a $29 update package.

  • #240 written by Bei October 28, 2014 (2 years ago)

    Hi,

    I canceled my TDS support plan many month ago and now I need it again (My license with TDS v1.3.2 only valid up to MT4 build 697 but I have 711 now).

    As I know there is a $29 update package to reactivate the TDS Support.
    Could you send me the link for it?

    Bei

    • #241 written by birt October 29, 2014 (2 years ago)

      I sent you an email with the link.

      • #242 written by Dennis March 31, 2016 (8 months ago)

        You have to change JForex interface language to English, when exporting ticks from JForex. In some other languages JForex use comma (,), not a dot (.) as fractional part delimiter, so prices are invalid.

  • #243 written by Selwyn November 3, 2014 (2 years ago)

    Hi,

    I have a EA which looks at multiple time periods (M1, D1 ect). When I optimize using tick data, can my EA look at the different time periods while optimizing? Is tick data M1 data, or is it all the periods in one? Thanks

    • #244 written by birt November 3, 2014 (2 years ago)

      A tick data FXT can be created for whatever timeframe you need to backtest your EA on. The CSV2FXT script will automatically generate HST files that are in sync with the FXT and will copy them to the correct folder so that during your backtest, any indicators or EA code accessing a timeframe other than the backtest timeframe will work correctly.

      Long story short, you can access different periods during a backtest with or without using tick data. The data is read from the HST files which contain bar data and the bar at index 0 has its low/high/close prices computed based on the current bid price.

      • #245 written by Selwyn November 4, 2014 (2 years ago)

        Thanks Birt. Do I need to change any settings in the CSV2FXT script while converting the tick data to get all the time periods? Or does the script convert all the time periods automatically? I get my tick data from Dukascopy.

        • #246 written by birt November 4, 2014 (2 years ago)

          No to #1.

          If you have any further support queries please address them to the support email address instead of posting here and please note that you need an active support subscription.

  • #247 written by Selwyn November 4, 2014 (2 years ago)

    Thanks Birt, but that’s all I need to know πŸ™‚

  • #248 written by Stefan December 13, 2014 (1 year ago)

    Hi Birt!

    I am using ur TDS for backtesting my EA that only works with Tick Data, no indicators or anything else are used. It is based on Tick Data performance infos like ticks per second ups and downs (transmitted to arrays and so on). But it doesn’t even show me same results one time.

    I am using MT4 Build 745, Duckascopy Tick Data M1.

    Do I missunderstood something on TDS?

    Thanks in advance for ur help!

    Cheers Stefan

    • #249 written by birt December 13, 2014 (1 year ago)

      If I understand correctly, if you click Start twice with the same EA and the same set of parameters, you obtain different results. Please correct me if I am wrong.

      This can typically have one of two reasons:
      1. You enabled slippage.
      2. Your EA uses random elements.

      To find out what is going on, please backtest the MACD EA to make sure that you obtain the same results in different runs. If you don’t, it’s slippage. If you do, it’s your EA.

      • #250 written by Stefan December 13, 2014 (1 year ago)

        Thanks for your fast response!

        Correct. I run the strategy tester twice or more with the same EA and the same set of parameters and I obtain different results. I don’t use random elements in the EA. The program checks SymbolInfoTick() and GetTickCount() to determine time and milli seconds only. Not random. Know what I do, I don’t be a beginner.

        Slippage… is this random in the MT test environment? Hmm… how to set the range…?

        I’ll make necessary tests, even with MACD EA if needed and respond.

        Thank you Birt!

        Cheers Stefan

        • #251 written by Stefan December 13, 2014 (1 year ago)

          slippage set to zero, same different results as before… run MACD EA tests tomorrow.

          • #252 written by birt December 14, 2014 (1 year ago)

            You should be aware that your EA introduces an element that varies between backtests which is the GetTickCount() function. This function returns the number of milliseconds since the system (your Windows machine) was started. It is completely useless in a backtest and most likely in regular trading as well, the only use I can think of being to measure performance.

            Consequently, if this is something that is somehow used in the decision process by the EA, the results will vary between runs because the number of milliseconds since the system start also varies.

            • #253 written by Stefan December 15, 2014 (1 year ago)

              Thank you for your help Birt!

              GetTickCount() was the problem. Maybe you know a possibility to determine the milliseconds between ticks? But many thanks to you for your help! I should read the manuel carefuly πŸ˜‰

              Cheers Stefan

              • #254 written by birt December 15, 2014 (1 year ago)

                MT4 backtesting does not have millisecond resolution. If two ticks occur in the same second, they will simply have the same timestamp (but they will of course come in chronological order).

  • #255 written by TDS testing result empty December 21, 2014 (1 year ago)

    Hi Birt,

    When I use TDS to test my EA, the test result is empty. Everything looks fine but no trades. I don`t see any error in the log. FXT file is also in place. I really don`t know why test result is empty. If I don`t use TDS to test, I can see some test result.

    Looking forward your reply. Thanks.

    • #256 written by TDS testing result empty December 21, 2014 (1 year ago)

      By the way, I don`t select optimization, just a simple test and not trades.

      • #257 written by birt December 21, 2014 (1 year ago)

        I suggest trying to backtest the MACD EA. If that one has trades, it’s either a problem that you introduced during the conversion process (e.g. spread too large, leverage too small etc.) or an issue with the EA itself. You could also try backtesting it in visual mode, perhaps that might offer more clues as to what is going on.

  • #258 written by Sam January 21, 2015 (1 year ago)

    Where i can see the variable spread in action?

  • #259 written by Sam January 21, 2015 (1 year ago)

    Where i can see the variable spread in action?
    IN VISUAL MODE with enabling the “Show Ask Line” i cannot see…

    • #260 written by birt January 22, 2015 (1 year ago)

      Create an EA that prints the difference between ask and bid on the chart and run it in visual mode. For example, Comment(“Spread: ” + DoubleToStr((Ask – Bid) / Point, 2));

      • #261 written by Gerard June 1, 2016 (6 months ago)

        Ok, I wondered about this also and printed MODE_SPREAD instead but did notice Ask – Bid was working, but still wasn’t 100% sure as it was printing out 1 for the spread, and changing the filter I had made no difference.
        I may have to make adjustments, to the ea as some trades filter by MODE_SPREAD but not a biggy.
        I must say though back testing this way is far superior at least for the tickscalper I am working on. I used to use tick story with the correct commissions and was getting around 3-4.5 profit factor, now I am getting around 2-3 but I guess is a lot more realistic.

  • #262 written by Rihan February 1, 2015 (1 year ago)

    Hi Birt,

    I did the following now am stuck.
    I downloaded the tick data with tickstory lite.
    Then I exported this to csv.
    Then I converted it with csv2fxt.
    In mt4 under the journals tab – it says “tickfile found”
    But then couple of minutes later it returns “no data”
    I went through the steps thoroughly – I have done this successfully in the past so don’t know where I am veering off. Please help.

    • #263 written by birt February 1, 2015 (1 year ago)

      Can you please send me an email (from the email address associated with your license) with the backtesting journal attached? Please also specify the length of your FXT file, the CSV2FXT version used and the MT4 build.

  • #264 written by Lee April 4, 2015 (1 year ago)

    Hi Brit,

    I’ve successfully converted PepperStone tick data into FXT using Pepperstone MT4 terminal. The latest few months chart time from Pepperstone is GMT +2, the converted data timing is GMT 0, therefore the 2 timing are no different. During conversion I use auto-detect for GMT, should I have set it differently?

    • #265 written by birt April 4, 2015 (1 year ago)

      Autodetect will do nothing in this case, likely it even printed a warning in the log about this. If you use autodetect for a data format that the script does not know, you will end up with a file that has the same GMT and DST as the input CSV.

      If you have any further support queries, whether you post them here or submit them via email (which is the preferred method) please use the address associated with your TDS license.

  • #266 written by ekke May 26, 2015 (1 year ago)

    Hi Birt

    I browsed your site with great interest.
    Does your suite incorporate the latest releases of MT4 (build 830) ?
    Must say I like your EA testing.

    regards
    ekke

    • #267 written by birt May 29, 2015 (1 year ago)

      There were a few difficulties (mostly the biggest problem was that I was out of town when it was released) but now it does, yes.

  • #268 written by Steve May 29, 2015 (1 year ago)

    2015.05.29 17:36:37.771 Forex-Flex-EA: no data for testing

    2015.05.29 17:36:37.771 TestGenerator: actual tick file “C:\Users\xxx\AppData\Roaming\MetaQuotes\Terminal\165A6291EBCD20797504A55466987405\tester\history\GBPUSDecn15_0.fxt” found

    I’m getting this no data for testing error on Tallinex-demo MT4. I’ve used tick data suite a lot a few years ago so I know what I’m doing. I did everything the same, downloading the tick data using strategy quant, converting etc… But I’ve never experienced this error before.

    It happens with or without the date selected. Changing the dates to anything has no affect either.

    What else should I look for?

    • #269 written by birt May 30, 2015 (1 year ago)

      Please update to Tick Data Suite v1.3.5. That problem is specific to v1.3.4 and MT4 builds 825 and 830.

  • #270 written by HERSON June 9, 2015 (1 year ago)

    Hi BIRT,

    I already located the CSV file (Which i download an export from SQ TickData Downloader) in the proper folder (MQL4/Files) And i re named it EURUSD.CSV

    But, when i run the CSV2FXT i get a message “Unable to understand date format” and then “Bad CSV format. Aborting”.

    Can you please helps me?

    Thanks

    • #271 written by birt June 9, 2015 (1 year ago)

      Sounds like you selected the wrong CSV format when exporting. Make sure the date and time are separated by a space and the fields are separated by a comma. Example:
      2014.01.01 22:00:58.616,1.37637,1.37553,0.1,0.1

      If you still can’t figure it out, please send me an email with the exact lines printed by CSV2FXT in your experts log.

      • #272 written by Herson June 10, 2015 (1 year ago)

        I open the CSV in Excel and effectively the date and time are separated by comas, not space. But in the configuring area of the SQ TickData Downloader there isn’t an option where i can change that way the CSV format separates the date and time.

        How can i solve it?

        • #273 written by birt June 10, 2015 (1 year ago)

          I’m not familiar with the SQ Tick Data Downloader but I’m sure there must be an option somewhere to export data in a format that’s good for CSV2FXT because many other people used it successfully. If you can’t find the option yourself, perhaps send a mail to SQ support.

          Another alternative would be to perform a regex replace on the whole file but unless you’re familiar with regex and have a program that can handle huge files, forget about it.

  • #274 written by Ivo June 18, 2015 (1 year ago)

    Hi Birt.

    I`m trying to test my EA on weekly chart. I know MT4 doesn’t support this time frame by default so I tried to import Weekly csv file to Daily chart using History tool in MT4. But doing this I get loads of mismatch chart errors as ST looks only on short time frame bars form 1 day not from all week as MT4 thinks its Daily chart.
    Is there a way to create weekly fxt file? And if yes would ST look at all the data from weekly chart not just the 1st day?

    Any help would be much appreciated.
    Anyone else tried testing using Weekly chart or other non-standard MT4 time frames?

    Thanks

    • #275 written by birt June 19, 2015 (1 year ago)

      As I explained via email, this is not possible without some additional fiddling. I recommend hardcoding the W1 timeframe in the EA where it applies and backtesting on a different timeframe.

      • #276 written by Ivo June 21, 2015 (1 year ago)

        Thanks for that. I did successfully change my EA to look at W1 timeframe when trading from any other time frame and it works nicely.
        Cheers

  • #277 written by Scott June 20, 2015 (1 year ago)

    Hey,

    Does your software have filters/cleaners/scrubbers for anomalous tick data (spikes) in the dukascopy BI5 files when exporting to csv/hst/fxt?

  • #280 written by Eddy July 9, 2015 (1 year ago)

    I am using mt4 v 845
    I did all steps, creat ftx files, start tds.exe

    but when I put inside the EA this code:
    Print(“Spread:”,DoubleToStr(MarketInfo(Symbol(),MODE_SPREAD)*Point,Digits));
    Print(“Ask:”,DoubleToStr(Ask,Digits),”, Bid:”,DoubleToStr(Bid,Digits),” Close0:”,DoubleToStr(Close[0],Digits));

    and start the backtest, I got the current spread only

    • #281 written by birt July 9, 2015 (1 year ago)

      Did you enable “Use Real Spread” in the CSV2FXT parameters?

    • #282 written by birt July 9, 2015 (1 year ago)

      Forgot to mention: MarketInfo(…, MODE_SPREAD) doesn’t work during backtests, it always prints a fixed spread. Use the difference between Ask and Bid.

      • #283 written by Bruno July 28, 2015 (1 year ago)

        Hi Birt, Is there a chance you can do something to fix that ?
        many experts advisors use now spread filters with MarketInfo(…, MODE_SPREAD)

        • #284 written by birt July 29, 2015 (1 year ago)

          I’ve tried before but unfortunately it’s not linked to the spread in the file (not even before build 840) which makes it quite hard to implement and especially maintain through new builds. I might take another look at it in the future.

  • #285 written by liudanbi August 3, 2015 (1 year ago)

    It shows: “Comma issues detected, perhaps extra field”

    • #286 written by birt August 3, 2015 (1 year ago)

      It means there is a problem with your CSV file.

  • #287 written by MIKE September 30, 2015 (1 year ago)

    good morning i have 890 build on fxpro and went thru this a ton have done same testing on other laptops numerous times my issue is after putting the csv file in proper place and starting the converter it box comes up and says i need to put the file in the proper place thank you in advance,

    • #288 written by birt September 30, 2015 (1 year ago)

      In MT4, please head to File -> Open Data Folder and verify that the CSV file is really in MQL4\Files. If it is, verify that the name is correct (e.g. EURUSD.csv for EURUSD unless you’re typing its name in the CSV Filename input) and that its extension is correct (e.g. .csv not .cvs or .csv.csv which can result from Windows Explorer hiding known file extensions). Please see http://eareview.net/tick-data/faq-troubleshooting#q-1-1 for more information.

      Don’t hesitate to send me an email if the above steps don’t solve your problem.

  • #289 written by Fxtech October 24, 2015 (1 year ago)

    I have MT4 build 890, and I tried doing a backtest and the data stops at October 6, 2015(AUDNZD), and October 2, 2015(EURUSD). I am using Tick Data Downloader, and have downloaded the same data 3 times to make sure it was not missing. I am using CSVed to look at the data and its all there. I thought it was the EA, so I tried the built in MACD EA , and still no data. Has anyone else had this issue?

    Thanks

    • #290 written by birt October 25, 2015 (1 year ago)

      Let me get this right: you’ve downloaded the data and you confirmed that the exported CSV has ticks up to 23.10.2015; then, you copied it to MQL4\Files and processed it with CSV2FXT once more and ran a backtest – is that correct? Did you check the exported CSV or the CSV in MQL4\Files?

      When running CSV2FXT it prints the last date in the file; can you check your expert logs to see what it printed when you converted? You answered “yes” when the conversion was over and you were asked whether you would like the script to move the files, right?

      Also, how did you determine that the data stops on those dates? By the date of the last trade or did you try a visual backtest with Use Date enabled?

      In any case, just for reference, nobody else reported a similar issue.

  • #291 written by ΧžΧ©Χ” December 17, 2015 (11 months ago)

    Good Day

    I purchased a possibility to do a test of 99%
    Now MetaTrader updated version of 940
    It does not work
    Do you have a solution for this
    I have a very urgent
    And I would be grateful if you could answer me
    Google translated so I hope that I will be understood

    • #292 written by birt December 17, 2015 (11 months ago)

      I replied extensively to the email you sent. It’s not necessary to post and also send an email; sending me an email is enough and usually that will get you a reply faster than posting here.

  • #293 written by Pavel January 12, 2016 (11 months ago)

    To every insane freak storing ΓΌber-quantum of FXTs on their hard drives:

    I’m doing a lot of testing recently, and I’ve got a LOT of FXTs (tick data from 4 sources: Dukascopy, Integral, FXOpen and LMAX). About 40 symbols in their respective folders, all timeframes. Additionaly to the “all-history” files, I tend to use split chunks for recent year and recent quarter in every folder – when working short term strategies, the tests start much faster without needing to skip to the start date, if the dataset was full… I bought a new 3TB drive three months ago; it was full in no time.

    I’m using remote Windows Server 2012 to run routine things, and Windows 10 at home for development. I have enabled data deduplication on both machines a week ago, and – still can’t believe it, though it’s very logical – my hard drives storing FXTs are almost empty now πŸ˜‰

    The reason: FXTs for different timeframes are virtually the same. And the smaller chunks are ‘included’ in the bigger ones. So the dedup saves space roughly by a factor of 10 πŸ™‚ Didn’t measure it, writes will almost certainly slower, but can’t notice any slowdown in testing! Really a great thing! Am doing IT stuff since decades, can’t remember to meet anything more useful, almost revolutionary.

    In order to run dedup on data volumes, you must have Windows 8 or higher – moreover, the feature has to be added (and managed) in a somewhat geeky manner to client version of Windows (8/8.1/10) via PowerShell… there are many guides on the Net… but it works as well as in server versions.

    • #294 written by Pavel January 12, 2016 (11 months ago)

      btw:
      hey Birt πŸ™‚
      thanks for inventing and maintaing the TDS – such a great thing πŸ™‚
      I’d have to abandon MT4 without it!
      https://i.imgur.com/LFtcMJ6.png

    • #295 written by birt January 12, 2016 (11 months ago)

      Just for reference, Windows data deduplication is not on the fly so the writes won’t be slower. It’s a scheduled process that typically runs at night and identifies and links blocks with similar content. I’m using it for my VMs for about 3 years now and yes, it’s awesome.

      Other than that, thanks for your kind comment πŸ™‚

  • #296 written by lee February 1, 2016 (10 months ago)

    using the CSV2FXT script (latest version) but even though I enter the spread parameter as 2.0, the spread is actually 0.0. I have checked the settings several times & have used the script many times previously successfully so something I am doing must be wrong?

    • #297 written by birt February 1, 2016 (10 months ago)

      That’s in all likelihood because you’re not using the Tick Data Suite. If that’s not the case, please send me an email from the email address associated with your support plan.

  • #298 written by Ben June 20, 2016 (5 months ago)

    Hi Birt, I’m using your php scripts to download dukascopy data for TDS. However this doesn’t work anymore since a few days: The cmd box says “Error code was 301”. Any idea?

    • #299 written by birt June 20, 2016 (5 months ago)

      What does it say other than that?

  • #300 written by Ben June 20, 2016 (5 months ago)

    It says:
    https://dl.dropboxusercontent.com/u/36606288/BirtScript_CMD_error.png
    Seems like Dukascopy changed the location where the data is stored…?

  • #303 written by John August 3, 2016 (4 months ago)

    Hi Birt,

    Thanks for your great software, it makes all the difference, and I ‘ve been happy to pay up for it! Don’t let it ever die, please…

    One question on .HST/.FXT files. I created rather large files based on the Pepperstone tick data. Conversion went fine, and tests run as expected.

    I would like to setup 2 instances of MT4 on my 2 Pepperstone accounts. Instead of doing the entire conversion through the CSV2FXT script twice (taking a long time on 2009-2016 data… several timeframes… several crosses πŸ˜‰ I was thinking of creating both first on MT4 instance no.1 and thereafter copying the .HST and .FXT files to their corresponding folders on MT4 instance no. 2 . In doing so, I’d prevent many hours in processing time.

    Is it possible to simply copy those files / re-use those files provided that I’m talking “same broker & account type” here ? Or would this leave MT4 no.2 in trouble nonetheless?

    Thanks in advance!
    John

    • #304 written by birt August 3, 2016 (4 months ago)

      Got no intention of letting it die, in fact I’m currently working on a major update that will take care of most of the issues you’re facing πŸ™‚

      Regarding your question, sure, it’s safe to copy the HST and FXT files from one MT4 to another. You could even run a quick MACD EA backtest in both terminals to ensure that you get identical results.

      Also note that if you want to run several backtests in parallel you can launch the same MT4 instance multiple times via TDS (simply double-click the shortcut as many times as you want), which makes duplicating the data useless (although you might have other valid reasons for having two instances with the same data).

  • #305 written by John August 4, 2016 (4 months ago)

    Great news, Birt rocks! πŸ™‚

    I had not realized I can run the MT4 instance multiple times through TDS, that is indeed an option in most cases.

    For one EA though, I will still need an MT4 copy for the original account, as the EA is coded in such a way that it requires me to run the System Tester on the live account linked to the EA (or otherwise I’d have to buy a licence for just a demo account number – which is not really an option πŸ˜‰ Yeah I know – that’s pretty lame, so I need to work around it a bit.

    Thanks & looking forward to your update. When is it planned?

    • #306 written by birt August 5, 2016 (4 months ago)

      If you need to backtest with your live account, I recommend making a copy of your MT4 folder and logging in with the investor password to avoid accidental trading. I would also recommend creating the FXT while connected to your live account to ensure that you have the min/max lot, lot step, swaps, stop distance, freeze level etc. from your broker.

      Regarding the new update, the beta should be available before the middle of the month. From there, due to the extensive changes (it’s essentially a complete rewrite) it might take anywhere between 2 to 8 weeks until the release.

  • #307 written by Ben August 16, 2016 (3 months ago)

    Hi Birt,
    a few posts above we already discussed the problem that Dukascopy changed the server location for the stored historic tick data. The solution was to replace “http://www.dukascopy.com” with “datafeed.dukascopy.com” in the respective PHP script. But now also the new address doesn’t work anymore. When I e.g. try to download data for GER30 – labeled “E_DAAX” in the script – I get the error code:
    “FATAL: Couldn’t download datafeed.dukascopy.com/E_DAAX/2012/00/19/16h_ticks.bid5.
    Error was: Empty reply from server”
    What should I do?
    Best,
    Ben

    • #308 written by birt August 17, 2016 (3 months ago)

      First of all, the correct download URL form is http://datafeed.dukascopy.com/datafeed/symbol/… – your path is missing the /datafeed/.

      Second, most of the indices and stocks symbols in the PHP have been either discontinued by Dukascopy or otherwise renamed. It is also the case of E_DAAX for which data no longer exists. The new symbol for the German 30 index is DEUIDXEUR. Tick data for it is available starting with 18.09.2011, 21:00. I will release an update soon to correct this but in the meantime I’m afraid you’re going to have to apply the fix manually.

      • #309 written by Ben August 17, 2016 (3 months ago)

        Yes, the “http://” must be included of course. But the problem was that “E_DAAX” has been renamed to “DEUIDXEUR” (as you explained). By the way: It seems that the history starts on 18.09.2012, 21:00 (and not 2011 as you said). The BI5 files before this date have almost all 0 KB. But also the BI5s since 18.09.2012, 21:00 seem to be incomplete because sometimes e.g. the “17h BI5” etc. has 0 KB. But this is a problem of Dukascopy of course and not an issue with your script.
        Best and thx,
        Ben

        • #310 written by birt August 17, 2016 (3 months ago)

          Keep in mind that some indices have trading hours so it’s normal to have no data for the greater part of a day.

  • #312 written by Ben October 20, 2016 (1 month ago)

    Hi Birt,
    First of all let me congratulate you on “TDS2”, it’s really an advancement! Currently I’m still dealing with “TDS1” and face(d) an issue that maybe also occurs in “TDS2”: The “Slippage Simulation Feature” works nice as long as the tested EA doesn’t use a “Max Deviation Feature” etc. to limit the maximum allowed slippage. But if an expert utilizes said “Max Deviation Feature” the TDS will show a curious behaviour in the backtest. If under said circumstances TDS will be applied on MT4 of a broker without offering a “Max Deviation (Slippage)”, the EA will skip all trades that would have a “wrong” entry price (slippage too big) although the broker only offers entries at the current market price with no exception.
    Even more strange: If TDS under said circumstances is applied on MT4 of a broker that is actually offering a “Max Deviation (Slippage)”, an expert (that utilizes said feature) will get price offers one after another within milliseconds until the “right” price (slippage not too big) is reached. But this of course doesn’t make any sense regarding real trading.
    Do you have any idea why TDS shows such a behaviour?
    Best,
    Ben

    • #313 written by birt October 20, 2016 (1 month ago)

      There’s nothing wrong with TDS, it produces slippage precisely in the limits you configure.

      However, note that you must configure the EA to also accept slippage within these limits, otherwise you will get lots of error 138 which is perfectly normal. Basically if the EA sends a max slippage with its order function calls and the simulated slippage is above that, you will get a requote – nothing unusual there.

  • #314 written by Guntis October 23, 2016 (1 month ago)

    Hello Birt,
    Congratulations on TDS2, It’s great!
    I have one little problem though. I downloaded “Germany 30” data with TDM, but MT4 strategy tester does not recognize the symbol, it has DE30 listed only. I can’t find the place to rename Germany 30 to DE30.
    Regards
    Guntis

    • #315 written by birt October 23, 2016 (1 month ago)

      You don’t have to rename anything. Open the Tick Data Settings dialog and select Germany 30 in the symbol dropdown.

      TDS does that automatically for Forex symbols but for indices and CFDs it’s impossible to cover all possible naming variants.

  • #316 written by Fran November 20, 2016 (2 weeks ago)

    Hi, I’m using TDS2. I’m trying to run multiple instances of mt4 in strategy tester. I can run multiple instances of mt4. I can run strategy tester in one instance, in the other ones strategy tester is waiting.

    How can i run multiple strategy tester?

    • #317 written by Fran November 20, 2016 (2 weeks ago)

      Hi Birt, sorry, i can do it without problem.

      • #318 written by birt November 21, 2016 (2 weeks ago)

        No problem.

        If you run into any other issues pleases use the support form instead of posting here.

  • You may use these HTML tags: <a> <abbr> <acronym> <b> <blockquote> <cite> <code> <del> <em> <i> <q> <s> <strike> <strong>

     

  • Comment Feed for this Post
Go to Top