Reading backtests critically: ten questions to ask before you trust a curve

A backtest report is the most persuasive document in automated trading and the easiest to manufacture. The MetaTrader strategy tester will produce a beautiful upward curve for almost any strategy if you let it choose the parameters, ignore the spread and test on the period the parameters were chosen from. That does not make backtesting useless; it means a report has to be read like evidence rather than admired like a picture. Here are the ten questions we ask before a vendor-provided backtest is labelled and published on a product page — and the ones you should ask too.

1. Which tester and which data?

The MT5 tester can use every real tick recorded by the broker (“Every tick based on real ticks”). The MT4 tester generates ticks from one-minute bars unless external tick data is imported, and reports a “modelling quality” percentage — 90% is the maximum for generated ticks and still means the intra-bar price path was invented. For strategies that hold trades for days, generated ticks are acceptable. For scalpers whose stops are a few pips wide, only real-tick testing tells you anything, because whether a stop was hit inside a one-minute bar is exactly what the model is guessing. The MT4 vs MT5 guide explains why most serious reports are now produced in MT5.

2. What spread and commission were used?

A fixed spread of one pip on EURUSD flatters almost every strategy. Real spreads widen at rollover, at news and during the Asian session, which are often precisely when a scalper is active. Look for “current spread” or, better, real-tick data that carries the historical spread, and check that commission per lot matches the account type the tool targets. A strategy that averages three pips of profit per trade is a different proposition when spread plus commission is two pips rather than half a pip.

3. How many trades?

Statistical confidence grows slowly with trade count. A report with 80 trades cannot distinguish a good strategy from luck, whatever the profit factor. As a rough guide, several hundred trades spread across different years and market conditions are the minimum before averages mean anything; a thousand or more is better. Also check the distribution: if 60% of the profit came from three trades, the strategy is not what the summary statistics describe.

4. Over what period, and does it include bad years?

Every strategy has market conditions it dislikes. A trend follower suffers in ranging years; a mean-reversion scalper suffers in sustained trends and during volatility shocks. A test that starts after the last difficult period for that style is not a test. Prefer reports that span at least five years and include at least one period you know was hostile to the strategy type. If the tool's author only shows the last eighteen months, ask why.

5. Were the parameters optimised on the same data?

This is the central problem. If the author ran an optimisation over 2019–2025 and then shows you a backtest over 2019–2025 with the winning parameters, the curve is fitted, not earned. The parameters have been chosen because they worked on that exact data. The honest approach is out-of-sample testing: optimise on one period, then test the chosen parameters on a later period the optimiser never saw. MT5's tester has a forward-testing option that does this automatically and reports both segments. A report that shows in-sample and out-of-sample results side by side is worth ten reports that show only a curve.

6. How many parameters, and how sensitive are they?

A strategy with twenty tunable inputs can fit anything. Ask how many inputs were optimised and whether neighbouring values also work. If a moving-average period of 47 is profitable but 45 and 49 lose money, the result is noise. Robust strategies show a plateau: a range of nearby parameter values with similar results. Authors who sell here are asked to document which inputs are meant to be adjusted and which should be left alone; a set file with dozens of finely tuned values is a warning sign, not a feature.

7. Balance curve or equity curve?

The default report chart shows balance, which only records closed trades. For strategies that hold or add to losers, the equity curve tells the real story. Open the equity line in the tester graph, or ask for the equity drawdown figure. A straight balance line with a deep equity valley beneath it describes a strategy that will one day fail to recover. Understanding drawdown explains why this distinction matters more than any other number in the report.

8. What sizing was used?

Fixed-lot testing and percentage-risk testing produce very different curves from the same trades. Compounding makes the later part of a curve look steep and the early drawdowns look trivial. Percentage figures for drawdown and return are only comparable when you know the sizing method, the starting balance and the leverage. If a report shows a 400% return with no mention of sizing, it tells you nothing about what the strategy earns per unit of risk.

9. Does the report match the product?

Check that the tested symbol, timeframe, account type and version match what is being sold. A report from version 2.1 on a hedging account says little about version 3.0 sold for netting accounts. On this site every performance record states the version and settings it belongs to; if a vendor's report cannot be tied to the current release, it is labelled accordingly or not published.

10. Is there any live or forward evidence?

A backtest plus several months of demo or live forward results with the same settings is far stronger than a backtest alone, because the forward period is out-of-sample by definition and includes real execution. Look for consistency between the two: similar trade frequency, similar average trade, similar behaviour around news. A live curve that looks nothing like the backtest means one of them is wrong, and it is usually the backtest.

How reports are labelled on this site

Every performance record on a product page carries one of four labels — vendor-provided, backtest, live-verified or demo — and backtests must state their tester, data quality, spread model, period, trade count and sizing. Sample listings created to demonstrate the site display figures marked “Sample data” that describe nothing. None of this makes a backtest a prediction; it makes it a document you can interrogate. Our full policy is on the Risk disclosure page.

Run your own

The best backtest is the one you run yourself with your broker's data, your account type and the set file you intend to use. Tools sold here include set files and documented inputs so that you can reproduce the author's test. If your result looks materially different from theirs, that is useful information — ask support which assumption differs before you trade it. A curve you cannot reproduce is a curve you should not rely on.