Precision for ta-lib 0.6.x was increased
but is lower by 0001 than what was in the freqtrade ta-lib fix
https://github.com/TA-Lib/ta-lib/blob/main/src/ta_func/ta_utility.h#L257
This won't be a problem for any normal usdt pair.
shitcoins on some exchanges (e.g gate) with 12 zeros before the first significant digit wouldn't have worked today, either.
Modify the original three duplicate functions (_profit_short, _profit_long, _profit), and add _profit_handler and _format_profit_message.
Refactor telegram.py and rpc.py.
Sorry for the duplicate functions yesterday, I was a bit rushed.
Both pytest and ruff have passed.
# Added `/profit_long` and `/profit_short` Commands
Users can now use commands like:
- `/profit_long [<n>]`
- `/profit_short [<n>]`
- `/profit [<n>]`
---
## Key Changes Implemented
### `freqtrade/rpc/telegram.py`:
- The `_profit` command handler has been updated to robustly parse `long` or `short` as optional arguments.
- **Translation:** The `_profit` command handler has been improved to reliably interpret `long` or `short` as optional parameters.
- The determined direction is passed to the RPC layer.
- **Translation:** The direction determined (either `long` or `short`) is passed to the RPC layer.
- The `/help` command documentation is updated.
- **Translation:** The documentation for the `/help` command has been updated accordingly.
---
### `freqtrade/rpc/rpc.py`:
- The `_rpc_trade_statistics` method now accepts a direction parameter.
- **Translation:** The `_rpc_trade_statistics` method has been updated to accept a `direction` parameter.
- The method has been refactored into a main function and a `_process_trade_stats` helper function to reduce complexity and improve readability.
- **Translation:** The method has been refactored into a main function and a helper function, `_process_trade_stats`, to reduce complexity and improve readability.
- The database query filter is dynamically modified to include a condition on `Trade.is_short` when a direction is provided.
- **Translation:** The database query filter dynamically adjusts to include a condition on `Trade.is_short` when a direction is specified.
---
### `tests/rpc/test_rpc_telegram.py`:
- Existing tests for `_profit` have been updated to match the new message format.
- **Translation:** Existing tests for the `_profit` function have been updated to match the new message format.
- New test cases have been added to specifically validate the `long` and `short` filtering functionality.
- **Translation:** New test cases have been added to specifically validate the filtering functionality for `long` and `short` trades.
---
## Testing
- All local `pytest` tests pass successfully.
- **Translation:** All local `pytest` tests have passed successfully.
- All `ruff` linter checks pass.
- **Translation:** All `ruff` code checks have passed.
- As I do not have a full local deployment, I am relying on the CI pipeline for final validation.
- **Translation:** Since I don't have a complete local deployment, I am relying on the CI pipeline for final validation.
---
This time, only a little AI was used :)
Except for the translation.
This commit enhances the /profit Telegram command to allow filtering by trade direction.
- The `_profit` handler in `telegram.py` now parses 'long'/'short' arguments and passes the direction to the RPC layer.
- The `_rpc_trade_statistics` method in `rpc.py` is updated to filter trades based on the provided direction. It has also been refactored for lower complexity.
- The `/help` command documentation is updated to reflect the new functionality.
- Corresponding unit tests in `test_rpc_telegram.py` are updated and extended to cover the new cases.
We'll for now issue a warning about this - and use the "current" tier
This way, gaps in tier data (between maxNotional and the next
minNotional) no longer cause an operational exception.
closes#11923
The bug happened since it just checked the length of the list itself, not what it represents. in this case .*/USDT could be any amount of pairs
when the user sets a max_open_trades of let's say 3 then the pairs only have 3 trade slots of whatever amount of pairs it really has and thereby creating a bottleneck.
This just sets the max_open_trades to -1 without even checking it, letting freqtrade itself handle the amount of trades allowed at a given time.
AI generated strategies and configs cause a lot of noise.
we should be clear that people shall read the documentation first
(this will also allow us to point people at this within the issue).
I got the following error on Pi 2 (using Debian Bookworm armhf arch)
```
...
running bdist_wheel
running build
running build_py
creating build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/lock.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/backend_ctypes.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_imp_emulation.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/model.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/ffiplatform.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/setuptools_ext.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/error.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/vengine_gen.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/api.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/__init__.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/recompiler.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/cffi_opcode.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/pkgconfig.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/verifier.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/cparser.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_shimmed_dist_utils.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/commontypes.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/vengine_cpy.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_cffi_include.h -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/parse_c_type.h -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_embedding.h -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_cffi_errors.h -> build/lib.linux-armv7l-cpython-311/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-armv7l-cpython-311/src/c
arm-linux-gnueabihf-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/freqtrade/.venv/include -I/usr/include/python3.11 -c src/c/_cffi_backend.c -o build/temp.linux-armv7l-cpython-311/src/c/_cffi_backend.o
src/c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
15 | #include <ffi.h>
| ^~~~~~~
compilation terminated.
error: command '/usr/bin/arm-linux-gnueabihf-gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for cffi
Failed to build cffi
ERROR: Failed to build installable wheels for some pyproject.toml based projects (cffi)
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Failed installing dependencies
```
It can be Easily solved by installing libffi-dev before hand
There are tests failing when using delayed and wrap_non_picklable_objects as decorator.
until I'll find a solution to run generate_optimizer standalone for analyze_per_epoch=True
with warnings.catch_warnings():
from optuna.exceptions import ExperimentalWarning
warnings.filterwarnings("ignore", category=FutureWarning)
this should be when importing sampler
This version supports numpy2 - while still remaining on ta-lib C of 0.4
This will avoid huge update problems, as the underlying library doesn't need to be updated
freqtrade/optimize/space/optunaspaces.py:39: error: Argument 1 to "__init__" of "IntDistribution" has incompatible type "int | float"; expected "int" [arg-type]
freqtrade/optimize/space/optunaspaces.py:39: error: Argument 2 to "__init__" of "IntDistribution" has incompatible type "int | float"; expected "int" [arg-type]
remove all references for ExtraTreesRegressor and skopt.space
Freqtrade is a free and open source crypto trading bot written in Python. It is designed to support all major exchanges and be controlled via Telegram or webUI. It contains backtesting, plotting and money management tools as well as strategy optimization by machine learning.
@@ -64,13 +63,12 @@ Please find the complete documentation on the [freqtrade website](https://www.fr
## Features
- [x]**Based on Python 3.10+**: For botting on any operating system - Windows, macOS and Linux.
- [x]**Based on Python 3.11+**: For botting on any operating system - Windows, macOS and Linux.
- [x]**Persistence**: Persistence is achieved through sqlite.
- [x]**Dry-run**: Run the bot without paying money.
- [x]**Backtesting**: Run a simulation of your buy/sell strategy.
- [x]**Strategy Optimization by machine learning**: Use machine learning to optimize your buy/sell strategy parameters with real exchange data.
- [X]**Adaptive prediction modeling**: Build a smart strategy with FreqAI that self-trains to the market via adaptive machine learning methods. [Learn more](https://www.freqtrade.io/en/stable/freqai/)
- [x]**Edge position sizing** Calculate your win rate, risk reward ratio, the best stoploss and adjust your position size before taking a position for each specific market. [Learn more](https://www.freqtrade.io/en/stable/edge/).
- [x]**Whitelist crypto-currencies**: Select which crypto-currency you want to trade or use dynamic whitelists.
- [x]**Blacklist crypto-currencies**: Select which crypto-currency you want to avoid.
- [x]**Builtin WebUI**: Builtin web UI to manage your bot.
@@ -112,7 +110,6 @@ positional arguments:
backtesting-show Show past Backtest results
backtesting-analysis
Backtest Analysis module.
edge Edge module.
hyperopt Hyperopt module.
hyperopt-list List Hyperopt results
hyperopt-show Show details of Hyperopt results
@@ -148,6 +145,8 @@ Telegram is not mandatory. However, this is a great way to control your bot. Mor
-`/stopentry`: Stop entering new trades.
-`/status <trade_id>|[table]`: Lists all or specific open trades.
-`/profit [<n>]`: Lists cumulative profit from all finished trades, over the last n days.
-`/profit_long [<n>]`: Lists cumulative profit from all finished long trades, over the last n days.
-`/profit_short [<n>]`: Lists cumulative profit from all finished short trades, over the last n days.
-`/forceexit <trade_id>|all`: Instantly exits the given trade (Ignoring `minimum_roi`).
-`/fx <trade_id>|all`: Alias to `/forceexit`
-`/performance`: Show performance of each finished trade grouped by pair
@@ -156,6 +155,7 @@ Telegram is not mandatory. However, this is a great way to control your bot. Mor
-`/help`: Show help message.
-`/version`: Show version.
## Development branches
The project is currently setup in two main branches:
@@ -221,7 +221,7 @@ To run this bot we recommend you a cloud instance with a minimum of:
"description":"Trading fee percentage. Can help to simulate slippage in backtesting",
"type":"number",
@@ -538,10 +548,6 @@
"description":"Exchange configuration.",
"$ref":"#/definitions/exchange"
},
"edge":{
"description":"Edge configuration.",
"$ref":"#/definitions/edge"
},
"log_config":{
"description":"Logging configuration.",
"$ref":"#/definitions/logging"
@@ -566,6 +572,7 @@
"pairlists":{
"description":"Configuration for pairlists.",
"type":"array",
"minItems":1,
"items":{
"type":"object",
"properties":{
@@ -610,11 +617,11 @@
"type":"string"
},
"chat_id":{
"description":"Telegram chat or group ID",
"description":"Telegram chat or group ID. Recommended to be set via environment variable FREQTRADE__TELEGRAM__CHAT_ID",
"type":"string"
},
"topic_id":{
"description":"Telegram topic ID - only applicable for group chats",
"description":"Telegram topic ID - only applicable for group chats. Recommended to be set via environment variable FREQTRADE__TELEGRAM__TOPIC_ID",
"type":"string"
},
"authorized_users":{
@@ -773,9 +780,11 @@
"type":"object",
"properties":{
"enabled":{
"description":"Enable webhook notifications.",
"type":"boolean"
},
"url":{
"description":"Webhook URL. Recommended to be set via environment variable FREQTRADE__WEBHOOK__URL",
"type":"string"
},
"format":{
@@ -853,6 +862,7 @@
"type":"boolean"
},
"webhook_url":{
"description":"Discord webhook URL. Recommended to be set via environment variable FREQTRADE__DISCORD__WEBHOOK_URL",
"type":"string"
},
"exit_fill":{
@@ -1168,28 +1178,35 @@
"description":"Name of the exchange.",
"type":"string"
},
"enable_ws":{
"description":"Enable WebSocket connections to the exchange.",
"type":"boolean",
"default":true
},
"key":{
"description":"API key for the exchange.",
"description":"API key for the exchange. Recommended to be set via environment variable FREQTRADE__EXCHANGE__KEY",
"type":"string",
"default":""
},
"secret":{
"description":"API secret for the exchange.",
"description":"API secret for the exchange. Recommended to be set via environment variable FREQTRADE__EXCHANGE__SECRET",
"type":"string",
"default":""
},
"password":{
"description":"Password for the exchange, if required.",
"description":"Password for the exchange, if required. Recommended to be set via environment variable FREQTRADE__EXCHANGE__PASSWORD",
"type":"string",
"default":""
},
"uid":{
"description":"User ID for the exchange, if required.",
"description":"User ID for the exchange, if required. Recommended to be set via environment variable FREQTRADE__EXCHANGE__UID",
"type":"string"
},
"account_id":{
"description":"Account ID for the exchange, if required. Recommended to be set via environment variable FREQTRADE__EXCHANGE__ACCOUNT_ID",
"type":"string"
},
"wallet_address":{
"description":"Wallet address for the exchange, if required. Usually used by DEX exchanges. Recommended to be set via environment variable FREQTRADE__EXCHANGE__WALLET_ADDRESS",
"type":"string"
},
"private_key":{
"description":"Private key for the exchange, if required. Usually used by DEX exchanges. Recommended to be set via environment variable FREQTRADE__EXCHANGE__PRIVATE_KEY",
"type":"string"
},
"pair_whitelist":{
@@ -1213,6 +1230,11 @@
"type":"boolean",
"default":false
},
"enable_ws":{
"description":"Enable WebSocket connections to the exchange.",
@@ -46,29 +46,32 @@ ranging from the simplest (0) to the most detailed per pair, per buy and per sel
More options are available by running with the `-h` option.
### Using export-filename
### Using backtest-filename
Normally, `backtesting-analysis` uses the latest backtest results, but if you wanted to go
back to a previous backtest output, you need to supply the `--export-filename` option.
You can supply the same parameter to `backtest-analysis` with the name of the final backtest
output file. This allows you to keep historical versions of backtest results and re-analyse
them at a later date:
By default, `backtesting-analysis` processes the most recent backtest results in the `user_data/backtest_results` directory.
If you want to analyze results from an earlier backtest, use the `--backtest-filename` option to specify the desired file. This lets you revisit and re-analyze historical backtest outputs at any time by providing the filename of the relevant backtest result:
Possible values are either one of "GP", "RF", "ET", "GBRT" (Details can be found in the [scikit-optimize documentation](https://scikit-optimize.github.io/)), or "an instance of a class that inherits from `RegressorMixin` (from sklearn) and where the `predict` method has an optional `return_std` argument, which returns `std(Y | x)` along with `E[Y | x]`".
Possible values are either one of "NSGAIISampler", "TPESampler", "GPSampler", "CmaEsSampler", "NSGAIIISampler", "QMCSampler" (Details can be found in the [optuna-samplers documentation](https://optuna.readthedocs.io/en/stable/reference/samplers/index.html)), or "an instance of a class that inherits from `optuna.samplers.BaseSampler`".
Some research will be necessary to find additional Regressors.
Example for `ExtraTreesRegressor` ("ET") with additional parameters:
# Corresponds to "ET" - but allows additional parameters.
return ExtraTreesRegressor(n_estimators=100)
```
The `dimensions` parameter is the list of `skopt.space.Dimension` objects corresponding to the parameters to be optimized. It can be used to create isotropic kernels for the `skopt.learning.GaussianProcessRegressor` estimator. Here's an example:
Some research will be necessary to find additional Samplers (from optunahub) for example.
!!! Note
While custom estimators can be provided, it's up to you as User to do research on possible parameters and analyze / understand which ones should be used.
If you're unsure about this, best use one of the Defaults (`"ET"` has proven to be the most versatile) without further parameters.
If you're unsure about this, best use one of the Defaults (`"NSGAIIISampler"` has proven to be the most versatile) without further parameters.
The syslog address can be either a Unix domain socket (socket filename) or a UDP socket specification, consisting of IP address and UDP port, separated by the `:` character.
So, the following are the examples of possible addresses:
* `"address": "/dev/log"` -- log to syslog (rsyslog) using the `/dev/log` socket, suitable for most systems.
@@ -323,20 +322,18 @@ So, the following are the examples of possible addresses:
* `"address": "localhost:514"` -- log to local syslog using UDP socket, if it listens on port 514.
* `"address": "<ip>:514"` -- log to remote syslog at IP address and port 514. This may be used on Windows for remote logging to an external syslog server.
??? Info "Deprecated - configure syslog via command line"
`--logfile syslog:<syslog_address>` -- send log messages to `syslog` service using the `<syslog_address>` as the syslog address.
`--logfile syslog:<syslog_address>` -- send log messages to `syslog` service using the `<syslog_address>` as the syslog address.
The syslog address can be either a Unix domain socket (socket filename) or a UDP socket specification, consisting of IP address and UDP port, separated by the `:` character.
The syslog address can be either a Unix domain socket (socket filename) or a UDP socket specification, consisting of IP address and UDP port, separated by the `:` character.
So, the following are the examples of possible usages:
So, the following are the examples of possible usages:
* `--logfile syslog:/dev/log` -- log to syslog (rsyslog) using the `/dev/log` socket, suitable for most systems.
* `--logfile syslog` -- same as above, the shortcut for `/dev/log`.
* `--logfile syslog:/var/run/syslog` -- log to syslog (rsyslog) using the `/var/run/syslog` socket. Use this on MacOS.
* `--logfile syslog:localhost:514` -- log to local syslog using UDP socket, if it listens on port 514.
* `--logfile syslog:<ip>:514` -- log to remote syslog at IP address and port 514. This may be used on Windows for remote logging to an external syslog server.
* `--logfile syslog:/dev/log` -- log to syslog (rsyslog) using the `/dev/log` socket, suitable for most systems.
* `--logfile syslog` -- same as above, the shortcut for `/dev/log`.
* `--logfile syslog:/var/run/syslog` -- log to syslog (rsyslog) using the `/var/run/syslog` socket. Use this on MacOS.
* `--logfile syslog:localhost:514` -- log to local syslog using UDP socket, if it listens on port 514.
* `--logfile syslog:<ip>:514` -- log to remote syslog at IP address and port 514. This may be used on Windows for remote logging to an external syslog server.
@@ -5,6 +5,8 @@ This page explains how to validate your strategy performance by using Backtestin
Backtesting requires historic data to be available.
To learn how to get data for the pairs and exchange you're interested in, head over to the [Data Downloading](data-download.md) section of the documentation.
Backtesting is also available in [webserver mode](freq-ui.md#backtesting), which allows you to run backtests via the web interface.
## Backtesting command reference
--8<-- "commands/backtesting.md"
@@ -103,12 +105,14 @@ Only use this if you're sure you'll not want to plot or analyze your results fur
---
Exporting trades to file specifying a custom filename
Exporting trades to file specifying a custom directory
The bot has made `429` trades for an average duration of `4:12:00`, with a performance of `76.20%` (profit), that means it has
earned a total of `0.00762792 BTC` starting with a capital of 0.01 BTC.
The bot has made `77` trades for an average duration of `22:12:00`, with a performance of `5.48%` (profit), that means it has earned a total of `54.774 USDT` starting with a capital of 1000 USDT.
The column `Avg Profit %` shows the average profit for all trades made.
The column `Tot Profit %` shows instead the total profit % in relation to the starting balance.
In the above results, we have a starting balance of 0.01 BTC and an absolute profit of 0.00762792 BTC - so the `Tot Profit %` will be `(0.00762792 / 0.01) * 100 ~= 76.2%`.
In the above results, we have a starting balance of 1000 USDT and an absolute profit of 54.774 USDT - so the `Tot Profit %` will be `(54.774 / 1000) * 100 ~= 5.48%`.
Your strategy performance is influenced by your entry strategy, your exit strategy, and also by the `minimal_roi` and `stop_loss` you have set.
@@ -282,85 +305,83 @@ On the other hand, if you set a too high `minimal_roi` like `"0": 0.55`
(55%), there is almost no chance that the bot will ever reach this profit.
Hence, keep in mind that your performance is an integral mix of all different elements of the strategy, your configuration, and the crypto-currency pairs you have set up.
### Exit reasons table
The 2nd table contains a recap of exit reasons.
This table can tell you which area needs some additional work (e.g. all or many of the `exit_signal` trades are losses, so you should work on improving the exit signal, or consider disabling it).
### Left open trades table
The 3rd table contains all trades the bot had to `force_exit` at the end of the backtesting period to present you the full picture.
The second table contains all trades the bot had to `force_exit` at the end of the backtesting period to present you the full picture.
This is necessary to simulate realistic behavior, since the backtest period has to end at some point, while realistically, you could leave the bot running forever.
These trades are also included in the first table, but are also shown separately in this table for clarity.
### Enter tag stats table
The third table provides a breakdown of trades by their entry tags (e.g., `enter_long`, `enter_short`), showing the number of entries, average profit percentage, total profit in the stake currency, total profit percentage, average duration, and the number of wins, draws, and losses for each tag.
### Exit reason stats table
The fourth table contains a recap of exit reasons (e.g., `exit_signal`, `roi`, `stop_loss`, `force_exit`). This table can tell you which area needs additional work (e.g., if many `exit_signal` trades are losses, you should work on improving the exit signal or consider disabling it).
### Mixed tag stats table
The fifth table combines entry tags and exit reasons, providing a detailed view of how different entry tags performed with specific exit reasons. This can help identify which combinations of entry and exit strategies are most effective.
### Summary metrics
The last element of the backtest report is the summary metrics table.
It contains some useful key metrics about performance of your strategy on backtesting data.
It contains key metrics about the performance of your strategy on backtesting data.
-`Backtesting from` / `Backtesting to`: Backtesting range (usually defined with the `--timerange` option).
-`Max open trades`: Setting of `max_open_trades` (or `--max-open-trades`) - or number of pairs in the pairlist (whatever is lower).
-`Trading Mode`: Spot or Futures trading.
-`Max open trades`: Setting of `max_open_trades` (or `--max-open-trades`) - or number of pairs in the pairlist (whatever is lower).
-`Total/Daily Avg Trades`: Identical to the total trades of the backtest output table / Total trades divided by the backtesting duration in days (this will give you information about how many trades to expect from the strategy).
-`Starting balance`: Start balance - as given by dry-run-wallet (config or command line).
-`Final balance`: Final balance - starting balance + absolute profit.
@@ -371,56 +392,71 @@ It contains some useful key metrics about performance of your strategy on backte
-`Sharpe`: Annualized Sharpe ratio.
-`Calmar`: Annualized Calmar ratio.
-`SQN`: System Quality Number (SQN) - by Van Tharp.
-`Profit factor`: profit / loss.
-`Profit factor`: Sum of the profits of all winning trades divided by the sum of the losses of all losing trades.
-`Expectancy (Ratio)`: Expectancy ratio, which is the average profit or loss per trade. A negative expectancy ratio means that your strategy is not profitable.
-`Avg. daily profit`: Average profit per day, calculated as `(Total Profit / Backtest Days)`.
-`Avg. stake amount`: Average stake amount, either `stake_amount` or the average when using dynamic stake amount.
-`Total trade volume`: Volume generated on the exchange to reach the above profit.
-`Best Pair` / `Worst Pair`: Best and worst performing pair, and it's corresponding `Tot Profit %`.
-`Best Trade` / `Worst Trade`: Biggest single winning trade and biggest single losing trade.
-`Long / Short trades`: Split long/short trade counts (only shown when short trades were made).
-`Long / Short profit %`: Profit percentage for long and short trades (only shown when short trades were made).
-`Long / Short profit USDT`: Profit in stake currency for long and short trades (only shown when short trades were made).
-`Best Pair` / `Worst Pair`: Best and worst performing pair (based on total profit percentage), and its corresponding `Tot Profit %`.
-`Best trade` / `Worst trade`: Biggest single winning trade and biggest single losing trade.
-`Best day` / `Worst day`: Best and worst day based on daily profit.
-`Days win/draw/lose`: Winning / Losing days (draws are usually days without closed trade).
-`Avg. Duration Winners` / `Avg. Duration Loser`: Average durations for winning and losing trades.
-`Days win/draw/lose`: Winning / Losing days (draws are usually days without closed trades).
-`Min/Max/Avg. Duration Winners`: Minimum, maximum, and average durations for winning trades.
-`Min/Max/Avg. Duration Losers`: Minimum, maximum, and average durations for losing trades.
-`Max Consecutive Wins / Loss`: Maximum consecutive wins/losses in a row.
-`Rejected Entry signals`: Trade entry signals that could not be acted upon due to `max_open_trades` being reached.
-`Entry/Exit Timeouts`: Entry/exit orders which did not fill (only applicable if custom pricing is used).
-`Canceled Trade Entries`: Number of trades that have been canceled by user request via `adjust_entry_price`.
-`Canceled Entry Orders`: Number of entry orders that have been canceled by user request via `adjust_entry_price`.
-`Replaced Entry Orders`: Number of entry orders that have been replaced by user request via `adjust_entry_price`.
-`Min balance` / `Max balance`: Lowest and Highest Wallet balance during the backtest period.
-`Max % of account underwater`: Maximum percentage your account has decreased from the top since the simulation started.
Calculated as the maximum of `(Max Balance - Current Balance) / (Max Balance)`.
-`Absolute Drawdown (Account)`: Maximum Account Drawdown experienced. Calculated as `(Absolute Drawdown) / (DrawdownHigh + startingBalance)`.
-`Drawdown`: Maximum, absolute drawdown experienced. Difference between Drawdown High and Subsequent Low point.
-`Drawdown high` / `Drawdown low`: Profit at the beginning and end of the largest drawdown period. A negative low value means initial capital lost.
-`Drawdown Start` / `Drawdown End`: Start and end datetime for this largest drawdown (can also be visualized via the `plot-dataframe` sub-command).
-`Market change`: Change of the market during the backtest period. Calculated as average of all pairs changes from the first to the last candle using the "close" column.
-`Long / Short`: Split long/short values (Only shown when short trades were made).
-`Total profit Long %` / `Absolute profit Long`: Profit long trades only (Only shown when short trades were made).
-`Total profit Short %` / `Absolute profit Short`: Profit short trades only (Only shown when short trades were made).
-`Max % of account underwater`: Maximum percentage your account has decreased from the top since the simulation started. Calculated as the maximum of `(Max Balance - Current Balance) / (Max Balance)`.
-`Absolute drawdown`: Maximum absolute drawdown experienced, including percentage relative to the account calculated as `(Absolute Drawdown) / (DrawdownHigh + startingBalance)`..
-`Drawdown duration`: Duration of the largest drawdown period.
-`Profit at drawdown start` / `Profit at drawdown end`: Profit at the beginning and end of the largest drawdown period.
-`Drawdown start` / `Drawdown end`: Start and end datetime for the largest drawdown (can also be visualized via the `plot-dataframe` sub-command).
-`Market change`: Change of the market during the backtest period. Calculated as the average of all pairs' changes from the first to the last candle using the "close" column.
### Daily / Weekly / Monthly breakdown
### Daily / Weekly / Monthly / Yearly breakdown
You can get an overview over daily / weekly or monthly results by using the `--breakdown <>` switch.
You can get an overview over daily, weekly, monthly, or yearly results by using the `--breakdown <>` switch.
To visualize daily and weekly breakdowns, you can use the following:
To visualize monthly and yearly breakdowns, you can use the following:
``` bash
freqtrade backtesting --strategy MyAwesomeStrategy --breakdown day week
freqtrade backtesting --strategy MyAwesomeStrategy --breakdown month year
```
``` output
======================== DAY BREAKDOWN =========================
The output will show a table containing the realized absolute Profit (in stake currency) for the given timeperiod, as well as wins, draws and losses that materialized (closed) on this day. Below that there will be a second table for the summarized values of weeks indicated by the date of the closing Sunday. The same would apply to a monthly breakdown indicated by the last day of the month.
The output will display tables containing the realized absolute profit (in stake currency) for the selected period, along with additional statistics such as number of trades, profit factor, and distribution of wins, draws, and losses that materialized (closed) on this period.
### Backtest result caching
@@ -435,15 +471,19 @@ To save time, by default backtest will reuse a cached result from within the las
To further analyze your backtest results, freqtrade will export the trades to file by default.
You can then load the trades to perform further analysis as shown in the [data analysis](strategy_analysis_example.md#load-backtest-results-to-pandas-dataframe) backtesting section.
Also, you can use freqtrade in [webserver mode](freq-ui.md#backtesting) to visualize the backtest results in a web interface.
This mode also allows you to load existing backtest results, so you can analyze them without running the backtest again.
For this mode - `--notes "<notes>"` can be used to add notes to the backtest results, which will be shown in the web interface.
### Backtest output file
The output file freqtrade produces is a zip file containing the following files:
- The backtest report in json format
- the market change data in feather format
- a copy of the strategy file
- a copy of the strategy parameters (if a parameter file was used)
- a sanitized copy of the config file
- The market change data in feather format
- A copy of the strategy file
- A copy of the strategy parameters (if a parameter file was used)
- A sanitized copy of the config file
This will ensure results are reproducible - under the assumption that the same data is available.
@@ -461,7 +501,7 @@ Since backtesting lacks some detailed information about what happens within a ca
- Exit-signal is favored over Stoploss, because exit-signals are assumed to trigger on candle's open
- ROI
- Exits are compared to high - but the ROI value is used (e.g. ROI = 2%, high=5% - so the exit will be at 2%)
- Exits are never "below the candle", so a ROI of 2% may result in a exit at 2.4% if low was at 2.4% profit
- Exits are never "below the candle", so a ROI of 2% may result in an exit at 2.4% if low was at 2.4% profit
- ROI entries which came into effect on the triggering candle (e.g. `120: 0.02` for 1h candles, from `60: 0.05`) will use the candle's open as exit rate
- Force-exits caused by `<N>=-1` ROI entries use low as exit value, unless N falls on the candle open (e.g. `120: -1` for 1h candles)
- Stoploss exits happen exactly at stoploss price, even if low was lower, but the loss will be `2 * fees` higher than the stoploss price
This will load 1h data (the main timeframe) as well as 5m data (detail timeframe) for the selected timerange.
The strategy will be analyzed with the 1h timeframe.
Candles where activity may take place (there's an active signal, the pair is in a trade) are evaluated at the 5m timeframe.
Candles where activity may take place (there's an active signal, the pair is in a trade) are evaluated at the 5m timeframe.
This will allow for a more accurate simulation of intra-candle movements - and can lead to different results, especially on higher timeframes.
Entries will generally still happen at the main candle's open, however freed trade slots may be freed earlier (if the exit signal is triggered on the 5m candle), which can then be used for a new trade of a different pair.
@@ -590,5 +630,5 @@ Detailed output for all strategies one after the other will be available, so mak
## Next step
Great, your strategy is profitable. What if the bot can give your the optimal parameters to use for your strategy?
Great, your strategy is profitable. What if the bot can give you the optimal parameters to use for your strategy?
Your next step is to learn [how to find optimal parameters with Hyperopt](hyperopt.md)
@@ -180,7 +180,7 @@ Mandatory parameters are marked as **Required**, which means that they are requi
| `minimal_roi` | **Required.** Set the threshold as ratio the bot will use to exit a trade. [More information below](#understand-minimal_roi). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Dict
| `stoploss` | **Required.** Value as ratio of the stoploss used by the bot. More details in the [stoploss documentation](stoploss.md). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Float (as ratio)
| `trailing_stop` | Enables trailing stoploss (based on `stoploss` in either configuration or strategy file). More details in the [stoploss documentation](stoploss.md#trailing-stop-loss). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Boolean
| `trailing_stop_positive` | Changes stoploss once profit has been reached. More details in the [stoploss documentation](stoploss.md#trailing-stop-loss-custom-positive-loss). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Float
| `trailing_stop_positive` | Changes stoploss once profit has been reached. More details in the [stoploss documentation](stoploss.md#trailing-stop-loss-different-positive-loss). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Float
| `trailing_stop_positive_offset` | Offset on when to apply `trailing_stop_positive`. Percentage value which should be positive. More details in the [stoploss documentation](stoploss.md#trailing-stop-loss-only-once-the-trade-has-reached-a-certain-offset). [Strategy Override](#parameters-in-the-strategy). <br>*Defaults to `0.0` (no offset).* <br> **Datatype:** Float
| `trailing_only_offset_is_reached` | Only apply trailing stoploss when the offset is reached. [stoploss documentation](stoploss.md). [Strategy Override](#parameters-in-the-strategy). <br>*Defaults to `false`.* <br> **Datatype:** Boolean
| `fee` | Fee used during backtesting / dry-runs. Should normally not be configured, which has freqtrade fall back to the exchange default fee. Set as ratio (e.g. 0.001 = 0.1%). Fee is applied twice for each trade, once when buying, once when selling. <br> **Datatype:** Float (as ratio)
@@ -234,7 +234,6 @@ Mandatory parameters are marked as **Required**, which means that they are requi
| `exchange.only_from_ccxt` | Prevent data-download from data.binance.vision. Leaving this as false can greatly speed up downloads, but may be problematic if the site is not available.<br>*Defaults to `false`*<br> **Datatype:** Boolean
| `experimental.block_bad_exchanges` | Block exchanges known to not work with freqtrade. Leave on default unless you want to test if that exchange works now. <br>*Defaults to `true`.* <br> **Datatype:** Boolean
| | **Plugins**
| `edge.*` | Please refer to [edge configuration document](edge.md) for detailed explanation of all possible configuration options.
| `pairlists` | Define one or more pairlists to be used. [More information](plugins.md#pairlists-and-pairlist-handlers). <br>*Defaults to `StaticPairList`.* <br> **Datatype:** List of Dicts
| | **Telegram**
| `telegram.enabled` | Enable the usage of Telegram. <br> **Datatype:** Boolean
@@ -567,8 +566,8 @@ Configuration:
### Understand order_time_in_force
The `order_time_in_force` configuration parameter defines the policy by which the order
is executed on the exchange. Three commonly used time in force are:
The `order_time_in_force` configuration parameter defines the policy by which the order is executed on the exchange.
Commonly used time in force are:
**GTC (Good Till Canceled):**
@@ -590,11 +589,13 @@ is automatically cancelled by the exchange.
Post only order. The order is either placed as a maker order, or it is canceled.
This means the order must be placed on orderbook for at least time in an unfilled state.
Please check the [Exchange documentation](exchanges.md) for supported time in force values for your exchange.
#### time_in_force config
The `order_time_in_force` parameter contains a dict with entry and exit time in force policy values.
This can be set in the configuration file or in the strategy.
Values set in the configuration file overwrites values set in the strategy.
Values set in the configuration file overwrite values from in the strategy, following the regular [precedence rules](#configuration-option-prevalence).
The possible values are: `GTC` (default), `FOK` or `IOC`.
@@ -606,9 +607,9 @@ The possible values are: `GTC` (default), `FOK` or `IOC`.
```
!!! Warning
This is ongoing work. For now, it is supported only for binance, gate and kucoin.
Please don't change the default value unless you know what you are doing and have researched the impact of using different values for your particular exchange.
### Fiat conversion
Freqtrade uses the Coingecko API to convert the coin value to it's corresponding fiat value for the Telegram reports.
@@ -672,7 +673,7 @@ Should you experience problems you suspect are caused by websockets, you can dis
}
```
Should you be required to use a proxy, please refer to the [proxy section](#using-proxy-with-freqtrade) for more information.
Should you be required to use a proxy, please refer to the [proxy section](#using-a-proxy-with-freqtrade) for more information.
!!! Info "Rollout"
We're implementing this out slowly, ensuring stability of your bots.
@@ -304,6 +304,13 @@ The `IProtection` parent class provides a helper method for this in `calculate_l
Most exchanges supported by CCXT should work out of the box.
If you need to implement a specific exchange class, these are found in the `freqtrade/exchange` source folder. You'll also need to add the import to `freqtrade/exchange/__init__.py` to make the loading logic aware of the new exchange.
We recommend looking at existing exchange implementations to get an idea of what might be required.
!!! Warning
Implementing and testing an exchange can be a lot of trial and error, so please bear this in mind.
You should also have some development experience, as this is not a beginner task.
To quickly test the public endpoints of an exchange, add a configuration for your exchange to `tests/exchange_online/conftest.py` and run these tests with `pytest --longrun tests/exchange_online/test_ccxt_compat.py`.
Completing these tests successfully a good basis point (it's a requirement, actually), however these won't guarantee correct exchange functioning, as this only tests public endpoints, but no private endpoint (like generate order or similar).
The `Edge Positioning` module uses probability to calculate your win rate and risk reward ratio. It will use these statistics to control your strategy trade entry points, position size and, stoploss.
!!! Danger "Deprecated functionality"
`Edge positioning` (or short Edge) is currently in maintenance mode only (we keep existing functionality alive) and should be considered as deprecated.
It will currently not receive new features until either someone stepped forward to take up ownership of that module - or we'll decide to remove edge from freqtrade.
!!! Warning
When using `Edge positioning` with a dynamic whitelist (VolumePairList), make sure to also use `AgeFilter` and set it to at least `calculate_since_number_of_days` to avoid problems with missing data.
!!! Note
`Edge Positioning` only considers *its own* buy/sell/stoploss signals. It ignores the stoploss, trailing stoploss, and ROI settings in the strategy configuration file.
`Edge Positioning` improves the performance of some trading strategies and *decreases* the performance of others.
## Introduction
Trading strategies are not perfect. They are frameworks that are susceptible to the market and its indicators. Because the market is not at all predictable, sometimes a strategy will win and sometimes the same strategy will lose.
To obtain an edge in the market, a strategy has to make more money than it loses. Making money in trading is not only about *how often* the strategy makes or loses money.
!!! tip "It doesn't matter how often, but how much!"
A bad strategy might make 1 penny in *ten* transactions but lose 1 dollar in *one* transaction. If one only checks the number of winning trades, it would be misleading to think that the strategy is actually making a profit.
The Edge Positioning module seeks to improve a strategy's winning probability and the money that the strategy will make *on the long run*.
We raise the following question[^1]:
!!! Question "Which trade is a better option?"
a) A trade with 80% of chance of losing 100\$ and 20% chance of winning 200\$<br/>
b) A trade with 100% of chance of losing 30\$
???+ Info "Answer"
The expected value of *a)* is smaller than the expected value of *b)*.<br/>
Hence, *b*) represents a smaller loss in the long run.<br/>
However, the answer is: *it depends*
Another way to look at it is to ask a similar question:
!!! Question "Which trade is a better option?"
a) A trade with 80% of chance of winning 100\$ and 20% chance of losing 200\$<br/>
b) A trade with 100% of chance of winning 30\$
Edge positioning tries to answer the hard questions about risk/reward and position size automatically, seeking to minimizes the chances of losing of a given strategy.
### Trading, winning and losing
Let's call $o$ the return of a single transaction $o$ where $o \in \mathbb{R}$. The collection $O = \{o_1, o_2, ..., o_N\}$ is the set of all returns of transactions made during a trading session. We say that $N$ is the cardinality of $O$, or, in lay terms, it is the number of transactions made in a trading session.
!!! Example
In a session where a strategy made three transactions we can say that $O = \{3.5, -1, 15\}$. That means that $N = 3$ and $o_1 = 3.5$, $o_2 = -1$, $o_3 = 15$.
A winning trade is a trade where a strategy *made* money. Making money means that the strategy closed the position in a value that returned a profit, after all deducted fees. Formally, a winning trade will have a return $o_i > 0$. Similarly, a losing trade will have a return $o_j \leq 0$. With that, we can discover the set of all winning trades, $T_{win}$, as follows:
$$ T_{win} = \{ o \in O | o > 0 \} $$
Similarly, we can discover the set of losing trades $T_{lose}$ as follows:
$$ T_{lose} = \{o \in O | o \leq 0\} $$
!!! Example
In a section where a strategy made four transactions $O = \{3.5, -1, 15, 0\}$:<br>
$T_{win} = \{3.5, 15\}$<br>
$T_{lose} = \{-1, 0\}$<br>
### Win Rate and Lose Rate
The win rate $W$ is the proportion of winning trades with respect to all the trades made by a strategy. We use the following function to compute the win rate:
$$W = \frac{|T_{win}|}{N}$$
Where $W$ is the win rate, $N$ is the number of trades and, $T_{win}$ is the set of all trades where the strategy made money.
Similarly, we can compute the rate of losing trades:
$$
L = \frac{|T_{lose}|}{N}
$$
Where $L$ is the lose rate, $N$ is the amount of trades made and, $T_{lose}$ is the set of all trades where the strategy lost money. Note that the above formula is the same as calculating $L = 1 – W$ or $W = 1 – L$
### Risk Reward Ratio
Risk Reward Ratio ($R$) is a formula used to measure the expected gains of a given investment against the risk of loss. It is basically what you potentially win divided by what you potentially lose. Formally:
$$ R = \frac{\text{potential_profit}}{\text{potential_loss}} $$
???+ Example "Worked example of $R$ calculation"
Let's say that you think that the price of *stonecoin* today is 10.0\$. You believe that, because they will start mining stonecoin, it will go up to 15.0\$ tomorrow. There is the risk that the stone is too hard, and the GPUs can't mine it, so the price might go to 0\$ tomorrow. You are planning to invest 100\$, which will give you 10 shares (100 / 10).
R &= \frac{\text{potential_profit}}{\text{potential_loss}}\\
&= \frac{50}{15}\\
&= 3.33
\end{aligned}$<br>
What it effectively means is that the strategy have the potential to make 3.33\$ for each 1\$ invested.
On a long horizon, that is, on many trades, we can calculate the risk reward by dividing the strategy' average profit on winning trades by the strategy' average loss on losing trades. We can calculate the average profit, $\mu_{win}$, as follows:
By combining the Win Rate $W$ and the Risk Reward ratio $R$ to create an expectancy ratio $E$. A expectance ratio is the expected return of the investment made in a trade. We can compute the value of $E$ as follows:
$$E = R * W - L$$
!!! Example "Calculating $E$"
Let's say that a strategy has a win rate $W = 0.28$ and a risk reward ratio $R = 5$. What this means is that the strategy is expected to make 5 times the investment around on 28% of the trades it makes. Working out the example:<br>
$E = R * W - L = 5 * 0.28 - 0.72 = 0.68$
<br>
The expectancy worked out in the example above means that, on average, this strategy' trades will return 1.68 times the size of its losses. Said another way, the strategy makes 1.68\$ for every 1\$ it loses, on average.
This is important for two reasons: First, it may seem obvious, but you know right away that you have a positive return. Second, you now have a number you can compare to other candidate systems to make decisions about which ones you employ.
It is important to remember that any system with an expectancy greater than 0 is profitable using past data. The key is finding one that will be profitable in the future.
You can also use this value to evaluate the effectiveness of modifications to this system.
!!! Note
It's important to keep in mind that Edge is testing your expectancy using historical data, there's no guarantee that you will have a similar edge in the future. It's still vital to do this testing in order to build confidence in your methodology but be wary of "curve-fitting" your approach to the historical data as things are unlikely to play out the exact same way for future trades.
## How does it work?
Edge combines dynamic stoploss, dynamic positions, and whitelist generation into one isolated module which is then applied to the trading strategy. If enabled in config, Edge will go through historical data with a range of stoplosses in order to find buy and sell/stoploss signals. It then calculates win rate and expectancy over *N* trades for each stoploss. Here is an example:
The goal here is to find the best stoploss for the strategy in order to have the maximum expectancy. In the above example stoploss at $3%$ leads to the maximum expectancy according to historical data.
Edge module then forces stoploss value it evaluated to your strategy dynamically.
### Position size
Edge dictates the amount at stake for each trade to the bot according to the following factors:
- Allowed capital at risk
- Stoploss
Allowed capital at risk is calculated as follows:
```
Allowed capital at risk = (Capital available_percentage) X (Allowed risk per trade)
```
Stoploss is calculated as described above with respect to historical data.
The position size is calculated as follows:
```
Position size = (Allowed capital at risk) / Stoploss
```
Example:
Let's say the stake currency is **ETH** and there is $10$ **ETH** on the wallet. The capital available percentage is $50%$ and the allowed risk per trade is $1\%$. Thus, the available capital for trading is $10 * 0.5 = 5$ **ETH** and the allowed capital at risk would be $5 * 0.01 = 0.05$ **ETH**.
-**Trade 1:** The strategy detects a new buy signal in the **XLM/ETH** market. `Edge Positioning` calculates a stoploss of $2\%$ and a position of $0.05 / 0.02 = 2.5$ **ETH**. The bot takes a position of $2.5$ **ETH** in the **XLM/ETH** market.
-**Trade 2:** The strategy detects a buy signal on the **BTC/ETH** market while **Trade 1** is still open. `Edge Positioning` calculates the stoploss of $4\%$ on this market. Thus, **Trade 2** position size is $0.05 / 0.04 = 1.25$ **ETH**.
!!! Tip "Available Capital $\neq$ Available in wallet"
The available capital for trading didn't change in **Trade 2** even with **Trade 1** still open. The available capital **is not** the free amount in the wallet.
-**Trade 3:** The strategy detects a buy signal in the **ADA/ETH** market. `Edge Positioning` calculates a stoploss of $1\%$ and a position of $0.05 / 0.01 = 5$ **ETH**. Since **Trade 1** has $2.5$ **ETH** blocked and **Trade 2** has $1.25$ **ETH** blocked, there is only $5 - 1.25 - 2.5 = 1.25$ **ETH** available. Hence, the position size of **Trade 3** is $1.25$ **ETH**.
!!! Tip "Available Capital Updates"
The available capital does not change before a position is sold. After a trade is closed the Available Capital goes up if the trade was profitable or goes down if the trade was a loss.
- The strategy detects a sell signal in the **XLM/ETH** market. The bot exits **Trade 1** for a profit of $1$ **ETH**. The total capital in the wallet becomes $11$ **ETH** and the available capital for trading becomes $5.5$ **ETH**.
-**Trade 4** The strategy detects a new buy signal int the **XLM/ETH** market. `Edge Positioning` calculates the stoploss of $2\%$, and the position size of $0.055 / 0.02 = 2.75$ **ETH**.
## Edge command reference
--8<-- "commands/edge.md"
## Configurations
Edge module has following configuration options:
| Parameter | Description |
|------------|-------------|
| `enabled` | If true, then Edge will run periodically. <br>*Defaults to `false`.* <br> **Datatype:** Boolean
| `process_throttle_secs` | How often should Edge run in seconds. <br>*Defaults to `3600` (once per hour).* <br> **Datatype:** Integer
| `calculate_since_number_of_days` | Number of days of data against which Edge calculates Win Rate, Risk Reward and Expectancy. <br> **Note** that it downloads historical data so increasing this number would lead to slowing down the bot. <br>*Defaults to `7`.* <br> **Datatype:** Integer
| `allowed_risk` | Ratio of allowed risk per trade. <br>*Defaults to `0.01` (1%)).* <br> **Datatype:** Float
| `stoploss_range_max` | Maximum stoploss. <br>*Defaults to `-0.10`.* <br> **Datatype:** Float
| `stoploss_range_step` | As an example if this is set to -0.01 then Edge will test the strategy for `[-0.01, -0,02, -0,03 ..., -0.09, -0.10]` ranges. <br> **Note** than having a smaller step means having a bigger range which could lead to slow calculation. <br> If you set this parameter to -0.001, you then slow down the Edge calculation by a factor of 10. <br>*Defaults to `-0.001`.* <br> **Datatype:** Float
| `minimum_winrate` | It filters out pairs which don't have at least minimum_winrate. <br>This comes handy if you want to be conservative and don't comprise win rate in favour of risk reward ratio. <br>*Defaults to `0.60`.* <br> **Datatype:** Float
| `minimum_expectancy` | It filters out pairs which have the expectancy lower than this number. <br>Having an expectancy of 0.20 means if you put 10\$ on a trade you expect a 12\$ return. <br>*Defaults to `0.20`.* <br> **Datatype:** Float
| `min_trade_number` | When calculating *W*, *R* and *E* (expectancy) against historical data, you always want to have a minimum number of trades. The more this number is the more Edge is reliable. <br>Having a win rate of 100% on a single trade doesn't mean anything at all. But having a win rate of 70% over past 100 trades means clearly something. <br>*Defaults to `10` (it is highly recommended not to decrease this number).* <br> **Datatype:** Integer
| `max_trade_duration_minute` | Edge will filter out trades with long duration. If a trade is profitable after 1 month, it is hard to evaluate the strategy based on it. But if most of trades are profitable and they have maximum duration of 30 minutes, then it is clearly a good sign.<br>**NOTICE:** While configuring this value, you should take into consideration your timeframe. As an example filtering out trades having duration less than one day for a strategy which has 4h interval does not make sense. Default value is set assuming your strategy interval is relatively small (1m or 5m, etc.).<br>*Defaults to `1440` (one day).* <br> **Datatype:** Integer
| `remove_pumps` | Edge will remove sudden pumps in a given market while going through historical data. However, given that pumps happen very often in crypto markets, we recommend you keep this off.<br>*Defaults to `false`.* <br> **Datatype:** Boolean
## Running Edge independently
You can run Edge independently in order to see in details the result. Here is an example:
Edge produced the above table by comparing `calculate_since_number_of_days` to `minimum_expectancy` to find `min_trade_number` historical information based on the config file. The timerange Edge uses for its comparisons can be further limited by using the `--timerange` switch.
In live and dry-run modes, after the `process_throttle_secs` has passed, Edge will again process `calculate_since_number_of_days` against `minimum_expectancy` to find `min_trade_number`. If no `min_trade_number` is found, the bot will return "whitelist empty". Depending on the trade strategy being deployed, "whitelist empty" may be return much of the time - or *all* of the time. The use of Edge may also cause trading to occur in bursts, though this is rare.
If you encounter "whitelist empty" a lot, condsider tuning `calculate_since_number_of_days`, `minimum_expectancy` and `min_trade_number` to align to the trading frequency of your strategy.
### Update cached pairs with the latest data
Edge requires historic data the same way as backtesting does.
Please refer to the [Data Downloading](data-download.md) section of the documentation for details.
Doing `--timerange=-20190901` will get all available data until September 1st (excluding September 1st 2019).
The full timerange specification:
* Use tickframes till 2018/01/31: `--timerange=-20180131`
* Use tickframes since 2018/01/31: `--timerange=20180131-`
* Use tickframes since 2018/01/31 till 2018/03/01 : `--timerange=20180131-20180301`
* Use tickframes between POSIX timestamps 1527595200 1527618600: `--timerange=1527595200-1527618600`
[^1]: Question extracted from MIT Opencourseware S096 - Mathematics with applications in Finance: https://ocw.mit.edu/courses/mathematics/18-s096-topics-in-mathematics-with-applications-in-finance-fall-2013/
Kucoin supports [time_in_force](configuration.md#understand-order_time_in_force) with settings "GTC" (good till cancelled), "FOK" (full-or-cancel) and "IOC" (immediate-or-cancel) settings.
!!! Tip "Stoploss on Exchange"
Kucoin supports `stoploss_on_exchange` and can use both stop-loss-market and stop-loss-limit orders. It provides great advantages, so we recommend to benefit from it.
@@ -271,7 +271,9 @@ Using the wrong exchange will result in the error "OKX Error 50119: API key does
## Gate.io
!!! Tip "Stoploss on Exchange"
Gate.io supports `stoploss_on_exchange` and uses `stop-loss-limit` orders. It provides great advantages, so we recommend to benefit from it by enabling stoploss on exchange..
Gate.io supports `stoploss_on_exchange` and uses `stop-loss-limit` orders. It provides great advantages, so we recommend to benefit from it by enabling stoploss on exchange.
Gate.io supports [time_in_force](configuration.md#understand-order_time_in_force) with settings "GTC" (good till cancelled), and "IOC" (immediate-or-cancel) settings.
Gate.io allows the use of `POINT` to pay for fees. As this is not a tradable currency (no regular market available), automatic fee calculations will fail (and default to a fee of 0).
The configuration parameter `exchange.unknown_fee_rate` can be used to specify the exchange rate between Point and the stake currency. Obviously, changing the stake-currency will also require changes to this value.
@@ -286,9 +288,15 @@ Without these permissions, the bot will not start correctly and show errors like
## Bybit
Futures trading on bybit is currently supported for USDT markets, and will use isolated futures mode.
!!! Tip "Stoploss on Exchange"
Bybit (futures only) supports `stoploss_on_exchange` and uses `stop-loss-limit` orders. It provides great advantages, so we recommend to benefit from it by enabling stoploss on exchange.
On futures, Bybit supports both `stop-limit` as well as `stop-market` orders. You can use either `"limit"` or `"market"` in the `order_types.stoploss` configuration setting to decide which type to use.
On startup, freqtrade will set the position mode to "One-way Mode" for the whole (sub)account. This avoids making this call over and over again (slowing down bot operations), but means that changes to this setting may result in exceptions and errors.
Bybit supports [time_in_force](configuration.md#understand-order_time_in_force) with settings "GTC" (good till cancelled), "FOK" (full-or-cancel), "IOC" (immediate-or-cancel) and "PO" (Post only) settings.
Futures trading on bybit is currently supported for isolated futures mode.
On startup, freqtrade will set the position mode to "One-way Mode" for the whole (sub)account. This avoids making this call over and over again (slowing down bot operations), but means that manual changes to this setting may result in exceptions and errors.
As bybit doesn't provide funding rate history, the dry-run calculation is used for live trades as well.
@@ -305,11 +313,6 @@ We do strongly recommend to limit all API keys to the IP you're going to use it
We therefore recommend the usage of one subaccount per bot. This is especially important when using unified accounts.
Other configurations (multiple bots on one account, manual non-bot trades on the bot account) are not supported and may lead to unexpected behavior.
!!! Tip "Stoploss on Exchange"
Bybit (futures only) supports `stoploss_on_exchange` and uses `stop-loss-limit` orders. It provides great advantages, so we recommend to benefit from it by enabling stoploss on exchange.
On futures, Bybit supports both `stop-limit` as well as `stop-market` orders. You can use either `"limit"` or `"market"` in the `order_types.stoploss` configuration setting to decide which type to use.
## Bitmart
Bitmart requires the API key Memo (the name you give the API key) to go along with the exchange key and secret.
@@ -328,6 +331,26 @@ It's therefore required to pass the UID as well.
!!! Warning "Necessary Verification"
Bitmart requires Verification Lvl2 to successfully trade on the spot market through the API - even though trading via UI works just fine with just Lvl1 verification.
## Bitget
Bitget requires a passphrase for each api key, you will therefore need to add this key into the configuration so your exchange section looks as follows:
```json
"exchange": {
"name": "bitget",
"key": "your_exchange_key",
"secret": "your_exchange_secret",
"password": "your_exchange_api_key_password",
// ...
}
```
Bitget supports [time_in_force](configuration.md#understand-order_time_in_force) with settings "GTC" (good till cancelled), "FOK" (full-or-cancel), "IOC" (immediate-or-cancel) and "PO" (Post only) settings.
!!! Tip "Stoploss on Exchange"
Bitget supports `stoploss_on_exchange` and can use both stop-loss-market and stop-loss-limit orders. It provides great advantages, so we recommend to benefit from it.
You can use either `"limit"` or `"market"` in the `order_types.stoploss` configuration setting to decide which type of stoploss shall be used.
## Hyperliquid
!!! Tip "Stoploss on Exchange"
@@ -339,13 +362,13 @@ This needs to be configured like this:
```json
"exchange": {
"name": "hyperliquid",
"walletAddress": "your_eth_wallet_address",
"walletAddress": "your_eth_wallet_address", // This should NOT be your API Wallet Address!
"privateKey": "your_api_private_key",
// ...
}
```
* walletAddress in hex format: `0x<40 hex characters>` - Can be easily copied from your wallet - and should be your wallet address, not your API Wallet Address.
* walletAddress in hex format: `0x<40 hex characters>` - Can be easily copied from your wallet - and should be your main wallet address, not your API Wallet Address.
* privateKey in hex format: `0x<64 hex characters>` - Use the key the API Wallet shows on creation.
Hyperliquid handles deposits and withdrawals on the Arbitrum One chain, a Layer 2 scaling solution built on top of Ethereum. Hyperliquid uses USDC as quote / collateral. The process of depositing USDC on Hyperliquid requires a couple of steps, see [how to start trading](https://hyperliquid.gitbook.io/hyperliquid-docs/onboarding/how-to-start-trading) for details on what steps are needed.
@@ -363,10 +386,50 @@ Hyperliquid handles deposits and withdrawals on the Arbitrum One chain, a Layer
* Create a different software wallet, only transfer the funds you want to trade with to that wallet, and use that wallet to trade on Hyperliquid.
* If you have funds you don't want to use for trading (after making a profit for example), transfer them back to your hardware wallet.
### Hyperliquid Vault / Subaccount
Hyperliquid allows you to create either a vault or a subaccount.
To use these with Freqtrade, you will need to use the following configuration pattern:
``` json
"exchange": {
"name": "hyperliquid",
"walletAddress": "your_vault_address", // Vault or subaccount address
"privateKey": "your_api_private_key",
"ccxt_config": {
"options": {
"vaultAddress": "your_vault_address" // Optional, only if you want to use a vault or subaccount
}
},
// ...
}
```
Your balance and trades will now be used from your vault / subaccount - and no longer from your main account.
### Historic Hyperliquid data
The Hyperliquid API does not provide historic data beyond the single call to fetch current data, so downloading data is not possible, as the downloaded data would not constitute proper historic data.
## Bitvavo
If your account is required to use an operatorId, you can set it in the configuration file as follows:
``` json
"exchange": {
"name": "bitvavo",
"key": "",
"secret": "",
"ccxt_config": {
"options": {
"operatorId": "123567"
}
},
}
```
Bitvavo expects the `operatorId` to be an integer.
## All exchanges
Should you experience constant errors with Nonce (like `InvalidNonce`), it is best to regenerate the API keys. Resetting Nonce is difficult and it's usually easier to regenerate the API keys.
@@ -102,6 +102,14 @@ You can use "current" market data by using the [dataprovider](strategy-customiza
You can use the `/stopentry` command in Telegram to prevent future trade entry, followed by `/forceexit all` (sell all open trades).
### I sold the bot's capital and now there's errors in the log
Freqtrade assumes that the trades it opens are managed only though the bot.
If you happen to (accidentally) sell the bot's capital, freqtrade will try to recover by trying to re-find on-exchange orders.
This is a best-effort approach, and will not work in all cases, especially when using order types that are not supported by freqtrade (OCO, iceberg, etc.), or when working with older trades (where the exchange no longer provides full order information).
The exact limits will vary between exchanges - with the details usually being documented in the exchange's API documentation.
### I want to run multiple bots on the same machine
Please look at the [advanced setup documentation Page](advanced-setup.md#running-multiple-instances-of-freqtrade).
@@ -151,6 +159,14 @@ This warning can point to one of the below problems:
* Barely traded pair -> Check the pair on the exchange webpage, look at the timeframe your strategy uses. If the pair does not have any volume in some candles (usually visualized with a "volume 0" bar, and a "_" as candle), this pair did not have any trades in this timeframe. These pairs should ideally be avoided, as they can cause problems with order-filling.
* API problem -> API returns wrong data (this only here for completeness, and should not happen with supported exchanges).
### I get the message "Couldn't reuse watch for xxx" in the log
This is an informational message that the bot tried to use candles from the websocket, but the exchange didn't provide the right information.
This can happen if there was an interruption to the websocket connection - or if the pair didn't have any trades happen in the timeframe you are using.
Freqtrade will handle this gracefully by falling back to the REST api.
While this makes the iteration slightly slower (due to the REST Api call) - it will not cause any problems to the bot's operation.
### I'm getting the "Exchange XXX does not support market orders." message and cannot run my strategy
As the message says, your exchange does not support market orders and you have one of the [order types](configuration.md/#understand-order_types) set to "market". Your strategy was probably written with other exchanges in mind and sets "market" orders for "stoploss" orders, which is correct and preferable for most of the exchanges supporting market orders (but not for Gate.io).
@@ -219,10 +235,7 @@ On Windows, the `--logfile` option is also supported by Freqtrade and you can us
First of all, most indicator libraries don't have GPU support - as such, there would be little benefit for indicator calculations.
The GPU improvements would only apply to pandas-native calculations - or ones written by yourself.
For hyperopt, freqtrade is using scikit-optimize, which is built on top of scikit-learn.
Their statement about GPU support is [pretty clear](https://scikit-learn.org/stable/faq.html#will-you-add-gpu-support).
GPU's also are only good at crunching numbers (floating point operations).
GPU's are only good at crunching numbers (floating point operations).
For hyperopt, we need both number-crunching (find next parameters) and running python code (running backtesting).
As such, GPU's are not too well suited for most parts of hyperopt.
@@ -271,20 +284,6 @@ Example: 4% profit 650 times vs 0,3% profit a trade 10000 times in a year. If we
### Edge implements interesting approach for controlling position size, is there any theory behind it?
The Edge module is mostly a result of brainstorming of [@mishaker](https://github.com/mishaker) and [@creslinux](https://github.com/creslinux) freqtrade team members.
You can find further info on expectancy, win rate, risk management and position size in the following sources:
@@ -181,7 +181,7 @@ You can ask for each of the defined features to be included also for informative
In total, the number of features the user of the presented example strategy has created is: length of `include_timeframes`* no. features in `feature_engineering_expand_*()` * length of `include_corr_pairlist` * no. `include_shifted_candles` * length of `indicator_periods_candles`
$= 3 * 3 * 3 * 2 * 2 = 108$.
!!! note "Learn more about creative feature engineering"
!!! note "Learn more about creative feature engineering"
Check out our [medium article](https://emergentmethods.medium.com/freqai-from-price-to-prediction-6fadac18b665) geared toward helping users learn how to creatively engineer features.
### Gain finer control over `feature_engineering_*` functions with `metadata`
@@ -310,7 +310,7 @@ class MyCoolTransform(BaseTransform):
If you have created your own custom `IFreqaiModel` with a custom `train()`/`predict()` function, *and* you still rely on `data_cleaning_train/predict()`, then you will need to migrate to the new pipeline. If your model does *not* rely on `data_cleaning_train/predict()`, then you do not need to worry about this migration.
More details about the migration can be found [here](strategy_migration.md#freqai---new-data-pipeline).
More details about the migration can be found [here](strategy_migration.md#freqai-new-data-pipeline).
If you are using docker, a dedicated tag with FreqAI dependencies is available as `:freqai`. As such - you can replace the image line in your docker compose file with `image: freqtradeorg/freqtrade:stable_freqai`. This image contains the regular FreqAI dependencies. Similar to native installs, Catboost will not be available on ARM based devices. If you would like to use PyTorch or Reinforcement learning, you should use the torch or RL tags, `image: freqtradeorg/freqtrade:stable_freqaitorch`, `image: freqtradeorg/freqtrade:stable_freqairl`.
!!! note "docker-compose-freqai.yml"
We do provide an explicit docker-compose file for this in `docker/docker-compose-freqai.yml` - which can be used via `docker compose -f docker/docker-compose-freqai.yml run ...` - or can be copied to replace the original docker file. This docker-compose file also contains a (disabled) section to enable GPU resources within docker containers. This obviously assumes the system has GPU resources available.
We do provide an explicit docker-compose file for this in `docker/docker-compose-freqai.yml` - which can be used via `docker compose -f docker/docker-compose-freqai.yml run ...` - or can be copied to replace the original docker file. This docker-compose file also contains a (disabled) section to enable GPU resources within docker containers. This obviously assumes the system has GPU resources available.
### FreqAI position in open-source machine learning landscape
This page explains how to tune your strategy by finding the optimal
parameters, a process called hyperparameter optimization. The bot uses algorithms included in the `scikit-optimize` package to accomplish this.
parameters, a process called hyperparameter optimization. The bot uses algorithms included in the `optuna` package to accomplish this.
The search will burn all your CPU cores, make your laptop sound like a fighter jet and still take a long time.
In general, the search for best parameters starts with a few random combinations (see [below](#reproducible-results) for more details) and then uses Bayesian search with a ML regressor algorithm (currently ExtraTreesRegressor) to quickly find a combination of parameters in the search hyperspace that minimizes the value of the [loss function](#loss-functions).
In general, the search for best parameters starts with a few random combinations (see [below](#reproducible-results) for more details) and then uses one of optuna's sampler algorithms (currently NSGAIIISampler) to quickly find a combination of parameters in the search hyperspace that minimizes the value of the [loss function](#loss-functions).
Hyperopt requires historic data to be available, just as backtesting does (hyperopt runs backtesting many times with different parameters).
To learn how to get data for the pairs and exchange you're interested in, head over to the [Data Downloading](data-download.md) section of the documentation.
The `-e` option will set how many evaluations hyperopt will do. Since hyperopt uses Bayesian search, running too many epochs at once may not produce greater results. Experience has shown that best results are usually not improving much after 500-1000 epochs.
The `--early-stop` option will set after how many epochs with no improvements hyperopt will stop. A good value is 20-30% of the total epochs. Any value greater than 0 and lower than 20 it will be replaced by 20. Early stop is by default disabled (`--early-stop=0`)
Doing multiple runs (executions) with a few 1000 epochs and different random state will most likely produce different results.
The `--spaces all` option determines that all possible parameters should be optimized. Possibilities are listed below.
@@ -532,7 +534,7 @@ Legal values are:
* `trailing`: search for the best trailing stop values
* `trades`: search for the best max open trades values
* `protection`: search for the best protection parameters (read the [protections section](#optimizing-protections) on how to properly define these)
* `default`: `all` except `trailing` and `protection`
* `default`: `all` except `trailing`, `trades` and `protection`
* space-separated list of any of the above values for example `--spaces roi stoploss`
The default Hyperopt Search Space, used when no `--space` command line option is specified, does not include the `trailing` hyperspace. We recommend you to run optimization for the `trailing` hyperspace separately, when the best parameters for other hyperspaces were found, validated and pasted into your custom strategy.
@@ -389,6 +389,8 @@ The `refresh_period` setting defines the interval (in seconds) at which the mark
The `categories` setting specifies the [coingecko categories](https://www.coingecko.com/en/categories) from which to select coins from. The default is an empty list `[]`, meaning no category filtering is applied.
If an incorrect category string is chosen, the plugin will print the available categories from CoinGecko and fail. The category should be the ID of the category, for example, for `https://www.coingecko.com/en/categories/layer-1`, the category ID would be `layer-1`. You can pass multiple categories such as `["layer-1", "meme-token"]` to select from several categories.
Coins like 1000PEPE/USDT or KPEPE/USDT:USDT are detected on a best effort basis, with the prefixes `1000` and `K` being used to identify them.
!!! Warning "Many categories"
Each added category corresponds to one API call to CoinGecko. The more categories you add, the longer the pairlist generation will take, potentially causing rate limit issues.
Freqtrade allows your strategy to implement different exit logic using signal-based or callback-based functions.
This section aims to compare each different function, helping you to choose the one that best fits your needs.
* **`populate_exit_trend()`** - Vectorized signal-based exit logic using indicators in the main dataframe
✅ **Use** to define exit signals based on indicators or other data that can be calculated in a vectorized manner.
🚫 **Don't use** to customize exit conditions for each individual trade, or if trade data is necessary to make an exit decision.
* **`custom_exit()`** - Custom exit logic that will fully exit a trade immediately, called for every open trade at every bot loop iteration until a trade is closed.
✅ **Use** to specify exit conditions for each individual trade (including any additional adjusted orders using `adjust_trade_position()`), or if trade data is necessary to make an exit decision, e.g. using profit data to exit.
🚫 **Don't use** when you want to exit using vectorised indicator-based data (use a `populate_exit_trend()` signal instead), or as a proxy for `custom_stoploss()`, and be aware that rate-based exits in backtesting can be inaccurate.
* **`custom_stoploss()`** - Custom trailing stoploss, called for every open trade every iteration until a trade is closed. The value returned here is also used for [stoploss on exchange](stoploss.md#stop-loss-on-exchangefreqtrade).
✅ **Use** to customize the stoploss logic to set a dynamic stoploss based on trade data or other conditions.
🚫 **Don't use** to exit a trade immediately based on a specific condition. Use `custom_exit()` for that purpose.
* **`custom_roi()`** - Custom ROI, called for every open trade every iteration until a trade is closed.
✅ **Use** to specify a minimum ROI threshold ("take-profit") to exit a trade at this ROI level at some point within the trade duration, based on profit or other conditions.
🚫 **Don't use** to exit a trade immediately based on a specific condition. Use `custom_exit()`.
🚫 **Don't use** for static ROI. Use `minimal_roi`.
@@ -31,7 +30,6 @@ Freqtrade is a free and open source crypto trading bot written in Python. It is
- Optimize: Find the best parameters for your strategy using hyperoptimization which employs machine learning methods. You can optimize buy, sell, take profit (ROI), stop-loss and trailing stop-loss parameters for your strategy.
- Select markets: Create your static list or use an automatic one based on top traded volumes and/or prices (not available during backtesting). You can also explicitly blacklist markets you don't want to trade.
- Run: Test your strategy with simulated money (Dry-Run mode) or deploy it with real money (Live-Trade mode).
- Run using Edge (optional module): The concept is to find the best historical [trade expectancy](edge.md#expectancy) by markets based on variation of the stop-loss and then allow/reject markets to trade. The sizing of the trade is based on a risk of a percentage of your capital.
- Control/Monitor: Use Telegram or a WebUI (start/stop the bot, show profit/loss, daily summary, current open trades results, etc.).
- Analyze: Further analysis can be performed on either Backtesting data or Freqtrade trading history (SQL database), including automated standard plots, and methods to load the data into [interactive environments](data-analysis.md).
@@ -88,7 +86,7 @@ To run this bot we recommend you a linux cloud instance with a minimum of:
@@ -24,7 +24,7 @@ The easiest way to install and run Freqtrade is to clone the bot Github reposito
The `stable` branch contains the code of the last release (done usually once per month on an approximately one week old snapshot of the `develop` branch to prevent packaging bugs, so potentially it's more stable).
!!! Note
Python3.10 or higher and the corresponding `pip` are assumed to be available. The install-script will warn you and stop if that's not the case. `git` is also needed to clone the Freqtrade repository.
Python3.11 or higher and the corresponding `pip` are assumed to be available. The install-script will warn you and stop if that's not the case. `git` is also needed to clone the Freqtrade repository.
Also, python headers (`python<yourversion>-dev` / `python<yourversion>-devel`) must be available for the installation to complete successfully.
!!! Warning "Up-to-date clock"
@@ -42,11 +42,10 @@ These requirements apply to both [Script Installation](#script-installation) and
This page explains how to validate your strategy in terms of lookahead bias.
This page explains how to validate your strategy in terms of lookahead bias.
Checking lookahead bias is the bane of any strategy since it is sometimes very easy to introduce backtest bias -
but very hard to detect.
Lookahead bias is the bane of any strategy since it is sometimes very easy to introduce this bias, but can be very hard to detect.
Backtesting initializes all timestamps at once and calculates all indicators in the beginning.
This means that if your indicators or entry/exit signals could look into future candles and falsify your backtest.
Backtesting initializes all timestamps (loads the whole dataframe into memory) and calculates all indicators at once.
This means that if your indicators or entry/exit signals look into future candles, this will falsify your backtest.
Lookahead-analysis requires historic data to be available.
The `lookahead-analysis` command requires historic data to be available.
To learn how to get data for the pairs and exchange you're interested in,
head over to the [Data Downloading](data-download.md) section of the documentation.
`lookahead-analysis` also supports freqai strategies.
This command is built upon backtesting since it internally chains backtests and pokes at the strategy to provoke it to show lookahead bias.
This is done by not looking at the strategy itself - but at the results it returned.
The results are things like changed indicator-values and moved entries/exits compared to the full backtest.
This command internally chains backtests and pokes at the strategy to provoke it to show lookahead bias.
This is done by not looking at the strategy code itself, but at changed indicator values and moved entries/exits compared to the full backtest.
You can use commands of [Backtesting](backtesting.md).
It also supports the lookahead-analysis of freqai strategies.
`lookahead-analysis` can use the typical options of [Backtesting](backtesting.md), but forces the following options:
- `--cache` is forced to "none".
- `--max-open-trades` is forced to be at least equal to the number of pairs.
@@ -25,48 +23,82 @@ It also supports the lookahead-analysis of freqai strategies.
- `--stake-amount` is forced to be a static 10000 (10k).
- `--enable-protections` is forced to be off.
Those are set to avoid users accidentally generating false positives.
These are set to avoid users accidentally generating false positives.
## Lookahead-analysis command reference
--8<-- "commands/lookahead-analysis.md"
!!! Note ""
The above Output was reduced to options `lookahead-analysis` adds on top of regular backtesting commands.
### Summary
Checks a given strategy for look ahead bias via lookahead-analysis
Look ahead bias means that the backtest uses data from future candles thereby not making it viable beyond backtesting
and producing false hopes for the one backtesting.
!!! Note
The above output was reduced to options that `lookahead-analysis` adds on top of regular backtesting commands.
### Introduction
Many strategies - without the programmer knowing - have fallen prey to lookahead bias.
Many strategies, without the programmer knowing, have fallen prey to lookahead bias.
This typically makes the strategy backtest look profitable, sometimes to extremes, but this is not realistic as the strategy is "cheating" by looking at data it would not have in dry or live modes.
Any backtest will populate the full dataframe including all timestamps at the beginning.
The reason why strategies can "cheat" is because the freqtrade backtesting process populates the full dataframe including all candle timestamps at the outset.
If the programmer is not careful or oblivious how things work internally
(which sometimes can be really hard to find out) then it will just look into the future making the strategy amazing
but not realistic.
(which sometimes can be really hard to find out) then the strategy will look into the future.
This command is made to try to verify the validity in the form of the aforementioned lookahead bias.
This command is made to try to verify the validity in the form of the aforementioned lookahead bias.
### How does the command work?
It will start with a backtest of all pairs to generate a baseline for indicators and entries/exits.
After the backtest ran, it will look if the `minimum-trade-amount` is met
and if not cancel the lookahead-analysis for this strategy.
After this initial backtest runs, it will look if the `minimum-trade-amount` is met and if not cancel the lookahead-analysis for this strategy.
If this happens, use a wider timerange to get more trades for the analysis, or use a timerange where more trades occur.
After setting the baseline it will then do additional runs for every entry and exit separately.
When a verification-backtest is done, it will compare the indicators as the signal (either entry or exit) and report the bias.
After all signals have been verified or falsified a result-table will be generated for the user to see.
After setting the baseline it will then do additional backtest runs for every entry and exit separately.
When these verificationbacktests complete, it will compare both dataframes (baseline and sliced) for any difference in columns' value and report the bias.
After all signals have been verified or falsified a resulttable will be generated for the user to see.
### How to find and remove bias? How can I salvage a biased strategy?
If you found a biased strategy online and want to have the same results, just without bias,
then you will be out of luck most of the time.
Usually the bias in the strategy is THE driving factor for "too good to be true" profits.
Removing conditions or indicators that push the profits up from bias will usually make the strategy significantly worse.
You might be able to salvage it partially if the biased indicators or conditions are not the core of the strategy, or there
are other entry and exit signals that are not biased.
### Examples of lookahead-bias
- `shift(-10)` looks 10 candles into the future.
- Using `iloc[]` in populate_* functions to access a specific row in the dataframe.
- For-loops are prone to introduce lookahead bias if you don't tightly control which numbers are looped through.
- Aggregation functions like `.mean()`, `.min()` and `.max()`, without a rolling window,
will calculate the value over the **whole** dataframe, so the signal candle will "see" a value including future candles.
A non-biased example would be to look back candles using `rolling()` instead:
e.g. `dataframe['volume_mean_12'] = dataframe['volume'].rolling(12).mean()`
- `ta.MACD(dataframe, 12, 26, 1)` will introduce bias with a signalperiod of 1.
### What do the columns in the results table mean?
- `filename`: name of the checked strategy file
- `strategy`: checked strategy class name
- `has_bias`: result of the lookahead-analysis. `No` would be good, `Yes` would be bad.
- `total_signals`: number of checked signals (default is 20)
- `biased_entry_signals`: found bias in that many entries
- `biased_exit_signals`: found bias in that many exits
- `biased_indicators`: shows you the indicators themselves that are defined in populate_indicators
You might get false positives in the `biased_exit_signals` if you have biased entry signals paired with those exits.
However, a biased entry will usually result in a biased exit too,
even if the exit itself does not produce the bias -
especially if your entry and exit conditions use the same biased indicator.
**Address the bias in the entries first, then address the exits.**
### Caveats
- `lookahead-analysis` can only verify / falsify the trades it calculated and verified.
If the strategy has many different signals / signal types, it's up to you to select appropriate parameters to ensure that all signals have triggered at least once. Not triggered signals will not have been verified.
This could lead to a false-negative (the strategy will then be reported as non-biased).
- `lookahead-analysis` has access to everything that backtesting has too.
Please don't provoke any configs like enabling position stacking.
If you decide to do so, then make doubly sure that you won't ever run out of `max_open_trades`amount and neither leftover money in your wallet.
- In the results table, the `biased_indicators` column will falsely flag FreqAI target indicators defined in `set_freqai_targets()` as biased. These are not biased and can safely be ignored.
If the strategy has many different signals / signal types, it's up to you to select appropriate parameters to ensure that all signals have triggered at least once. Signals that are not triggered will not have been verified.
This would lead to a false-negative, i.e. the strategy will be reported as non-biased.
- `lookahead-analysis` has access to the same backtesting options and this can introduce problems.
Please don't use any options like enabling position stacking as this will distort the number of checked signals.
If you decide to do so, then make doubly sure that you won't ever run out of `max_open_trades`slots,
and that you have enough capital in the backtest wallet configuration.
- In the results table, the `biased_indicators` column
will falsely flag FreqAI target indicators defined in `set_freqai_targets()` as biased.
**These are not biased and can safely be ignored.**
@@ -50,6 +50,7 @@ Enable subscribing to an instance by adding the `external_message_consumer` sect
| `ping_timeout` | Ping timeout <br>*Defaults to `10`.*<br> **Datatype:** Integer - in seconds.
| `sleep_time` | Sleep time before retrying to connect.<br>*Defaults to `10`.*<br> **Datatype:** Integer - in seconds.
| `remove_entry_exit_signals` | Remove signal columns from the dataframe (set them to 0) on dataframe receipt.<br>*Defaults to `false`.*<br> **Datatype:** Boolean.
| `initial_candle_limit` | Initial candles to expect from the Producer.<br>*Defaults to `1500`.*<br> **Datatype:** Integer - Number of candles.
| `message_size_limit` | Size limit per message<br>*Defaults to `8`.*<br> **Datatype:** Integer - Megabytes.
Instead of (or as well as) calculating indicators in `populate_indicators()` the follower instance listens on the connection to a producer instance's messages (or multiple producer instances in advanced configurations) and requests the producer's most recently analyzed dataframes for each pair in the active whitelist.
:param trade_id: Deletes the trade with this ID from the database.
edge
Return information about edge.
forcebuy
Buy an asset.
@@ -368,7 +365,6 @@ All endpoints in the below table need to be prefixed with the base URL of the AP
| `/blacklist` | GET | Show the current blacklist.
| `/blacklist` | POST | Adds the specified pair to the blacklist.<br/>*Params:*<br/>- `pair` (`str`)
| `/blacklist` | DELETE | Deletes the specified list of pairs from the blacklist.<br/>*Params:*<br/>- `[pair,pair]` (`list[str]`)
| `/edge` | GET | Show validated pairs by Edge if it is enabled.
| `/pair_candles` | GET | Returns dataframe for a pair / timeframe combination while the bot is running. **Alpha**
| `/pair_candles` | POST | Returns dataframe for a pair / timeframe combination while the bot is running, filtered by a provided list of columns to return. **Alpha**<br/>*Params:*<br/>- `<column_list>` (`list[str]`)
| `/pair_history` | GET | Returns an analyzed dataframe for a given timerange, analyzed by a given strategy. **Alpha**
@@ -31,6 +31,7 @@ The Order-type will be ignored if only one mode is available.
| Binance | limit |
| Binance Futures | market, limit |
| Bingx | market, limit |
| Bitget | market, limit |
| HTX | limit |
| kraken | market, limit |
| Gate | limit |
@@ -256,4 +257,4 @@ The new stoploss value will be applied to open trades (and corresponding log-mes
### Limitations
Stoploss values cannot be changed if `trailing_stop` is enabled and the stoploss has already been adjusted, or if [Edge](edge.md) is enabled (since Edge would recalculate stoploss based on the current market situation).
Stoploss values cannot be changed if `trailing_stop` is enabled and the stoploss has already been adjusted.
A simple callback which is called once when the strategy is loaded.
@@ -121,7 +125,7 @@ Freqtrade will fall back to the `proposed_stake` value should your code raise an
Called for open trade every throttling iteration (roughly every 5 seconds) until a trade is closed.
Allows to define custom exit signals, indicating that specified position should be sold. This is very useful when we need to customize exit conditions for each individual trade, or if you need trade data to make an exit decision.
Allows to define custom exit signals, indicating that specified position should be closed (full exit). This is very useful when we need to customize exit conditions for each individual trade, or if you need trade data to make an exit decision.
For example you could implement a 1:2 risk-reward ROI with `custom_exit()`.
@@ -178,6 +182,8 @@ Returning `None` will be interpreted as "no desire to change", and is the only s
Stoploss on exchange works similar to `trailing_stop`, and the stoploss on exchange is updated as configured in `stoploss_on_exchange_interval` ([More details about stoploss on exchange](stoploss.md#stop-loss-on-exchangefreqtrade)).
If you're on futures markets, please take note of the [stoploss and leverage](stoploss.md#stoploss-and-leverage) section, as the stoploss value returned from `custom_stoploss` is the risk for this trade - not the relative price movement.
!!! Note "Use of dates"
All time-based calculations should be done based on `current_time` - using `datetime.now()` or `datetime.utcnow()` is discouraged, as this will break backtesting support.
@@ -233,7 +239,7 @@ class AwesomeStrategy(IStrategy):
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
:return float: New stoploss value, relative to the current_rate
"""
return -0.04
return -0.04 * trade.leverage
```
#### Time based trailing stop
@@ -255,9 +261,9 @@ class AwesomeStrategy(IStrategy):
# Make sure you have the longest interval first - these conditions are evaluated from top to bottom.
if current_time - timedelta(minutes=120) > trade.open_date_utc:
@@ -497,6 +503,135 @@ The helper function `stoploss_from_absolute()` can be used to convert from an ab
---
## Custom ROI
Called for open trade every iteration (roughly every 5 seconds) until a trade is closed.
The usage of the custom ROI method must be enabled by setting `use_custom_roi=True` on the strategy object.
This method allows you to define a custom minimum ROI threshold for exiting a trade, expressed as a ratio (e.g., `0.05` for 5% profit). If both `minimal_roi` and `custom_roi` are defined, the lower of the two thresholds will trigger an exit. For example, if `minimal_roi` is set to `{"0": 0.10}` (10% at 0 minutes) and `custom_roi` returns `0.05`, the trade will exit when the profit reaches 5%. Also, if `custom_roi` returns `0.10` and `minimal_roi` is set to `{"0": 0.05}` (5% at 0 minutes), the trade will be closed when the profit reaches 5%.
The method must return a float representing the new ROI threshold as a ratio, or `None` to fall back to the `minimal_roi` logic. Returning `NaN` or `inf` values is considered invalid and will be treated as `None`, causing the bot to use the `minimal_roi` configuration.
### Custom ROI examples
The following examples illustrate how to use the `custom_roi` function to implement different ROI logics.
#### Custom ROI per side
Use different ROI thresholds depending on the `side`. In this example, 5% for long entries and 2% for short entries.
By default, freqtrade use the orderbook to automatically set an order price([Relevant documentation](configuration.md#prices-used-for-orders)), you also have the option to create custom order prices based on your strategy.
@@ -1107,3 +1242,119 @@ class AwesomeStrategy(IStrategy):
return None
```
## Plot annotations callback
The plot annotations callback is called whenever freqUI requests data to display a chart.
This callback has no meaning in the trade cycle context and is only used for charting purposes.
The strategy can then return a list of `AnnotationType` objects to be displayed on the chart.
Depending on the content returned - the chart can display horizontal areas, vertical areas, or boxes.
The full object looks like this:
``` json
{
"type": "area", // Type of the annotation, currently only "area" is supported
"start": "2024-01-01 15:00:00", // Start date of the area
"end": "2024-01-01 16:00:00", // End date of the area
"y_start": 94000.2, // Price / y axis value
"y_end": 98000, // Price / y axis value
"color": "",
"label": "some label"
}
```
The below example will mark the chart with areas for the hours 8 and 15, with a grey color, highlighting the market open and close hours.
@@ -229,6 +229,7 @@ official commands. You can ask at any moment for help with `/help`.
| `/cancel_open_order <trade_id> | /coo <trade_id>` | Cancel an open order for a trade.
| **Metrics** |
| `/profit [<n>]` | Display a summary of your profit/loss from close trades and some stats about your performance, over the last n days (all trades by default)
| `/profit_[long|short] [<n>]` | Display a summary of your profit/loss from close trades in one direction and some stats about your performance, over the last n days (all trades by default)
| `/performance` | Show performance of each finished trade grouped by pair
| `/balance` | Show bot managed balance per currency
| `/balance full` | Show account balance per currency
@@ -240,7 +241,6 @@ official commands. You can ask at any moment for help with `/help`.
| `/entries` | Shows Wins / losses by Exit reason as well as Avg. holding durations for buys and sells
| `/whitelist [sorted] [baseonly]` | Show the current whitelist. Optionally display in alphabetical order and/or with just the base currency of each pairing.
| `/blacklist [pair]` | Show the current blacklist, or adds a pair to the blacklist.
| `/edge` | Show validated pairs by Edge if it is enabled.
## Telegram commands in action
@@ -310,6 +310,8 @@ current max
### /profit
Also available as `/profit_long` and `/profit_short` to show profit for long or short trades only.
Return a summary of your profit/loss and performance.
> **ROI:** Close trades
@@ -451,21 +453,6 @@ Use `/reload_config` to reset the blacklist.
> Using blacklist `StaticPairList` with 2 pairs
>`DODGE/BTC`, `HOT/BTC`.
### /edge
Shows pairs validated by Edge along with their corresponding win-rate, expectancy and stoploss values.
@@ -134,15 +134,17 @@ Most properties here can be None as they are dependent on the exchange response.
|------------|-------------|-------------|
| `trade` | Trade | Trade object this order is attached to |
| `ft_pair` | string | Pair this order is for |
| `ft_is_open` | boolean | is the order filled? |
| `ft_is_open` | boolean | is the order still open? |
| `order_type` | string | Order type as defined on the exchange - usually market, limit or stoploss |
| `status` | string | Status as defined by ccxt. Usually open, closed, expired or canceled |
| `side` | string | Buy or Sell |
| `status` | string | Status as defined by [ccxt's order structure](https://docs.ccxt.com/#/README?id=order-structure). Usually open, closed, expired, canceled or rejected |
| `side` | string | buy or sell |
| `price` | float | Price the order was placed at |
| `average` | float | Average price the order filled at |
| `amount` | float | Amount in base currency |
| `filled` | float | Filled amount (in base currency) |
| `remaining` | float | Remaining amount |
| `filled` | float | Filled amount (in base currency) (use `safe_filled` instead) |
| `safe_filled` | float | Filled amount (in base currency) - guaranteed to not be None |
| `safe_remaining` | float | Remaining amount - either taken from the exchange or calculated. |
| `cost` | float | Cost of the order - usually average * filled (*Exchange dependent on futures, may contain the cost with or without leverage and may be in contracts.*) |
| `stake_amount` | float | Stake amount used for this order. *Added in 2023.7.* |
| `stake_amount_filled` | float | Filled Stake amount used for this order. *Added in 2024.11.* |
Update-problems usually come missing dependencies (you didn't follow the above instructions) - or from updated dependencies, which fail to install (for example TA-lib).
Please refer to the corresponding installation sections (common problems linked below)
Common problems and their solutions:
* [ta-lib update on windows](windows_installation.md#2-install-ta-lib)
Besides the Live-Trade and Dry-Run run modes, the `backtesting`, `edge` and `hyperopt` optimization subcommands, and the `download-data` subcommand which prepares historical data, the bot contains a number of utility subcommands. They are described in this section.
Besides the Live-Trade and Dry-Run run modes, the `backtesting` and `hyperopt` optimization subcommands, and the `download-data` subcommand which prepares historical data, the bot contains a number of utility subcommands. They are described in this section.
@@ -117,9 +117,9 @@ Different payloads can be configured for different events. Not all fields are ne
## Webhook Message types
### Entry
### Entry / Entry fill
The fields in `webhook.entry` are filled when the bot executes a long/short. Parameters are filled using string.format.
The fields in `webhook.entry` and `webhook.entry_fill` are filled when the bot places a long/short Order to increase a position, or when that order fills respectively. Parameters are filled using string.format.
Possible parameters are:
* `trade_id`
@@ -162,31 +162,9 @@ Possible parameters are:
* `current_rate`
* `enter_tag`
### Entry fill
### Exit / Exit fill
The fields in `webhook.entry_fill` are filled when the bot filled a long/short order. Parameters are filled using string.format.
Possible parameters are:
* `trade_id`
* `exchange`
* `pair`
* `direction`
* `leverage`
* `open_rate`
* `amount`
* `open_date`
* `stake_amount`
* `stake_currency`
* `base_currency`
* `quote_currency`
* `fiat_currency`
* `order_type`
* `current_rate`
* `enter_tag`
### Exit
The fields in `webhook.exit` are filled when the bot exits a trade. Parameters are filled using string.format.
The fields in `webhook.exit` and `webhook.exit_fill` are filled when the bot places an exit order, or when that exit order fills respectively. Parameters are filled using string.format.
Possible parameters are:
* `trade_id`
@@ -195,34 +173,9 @@ Possible parameters are:
* `direction`
* `leverage`
* `gain`
* `limit`
* `amount`
* `open_rate`
* `profit_amount`
* `profit_ratio`
* `stake_currency`
* `base_currency`
* `quote_currency`
* `fiat_currency`
* `exit_reason`
* `order_type`
* `open_date`
* `close_date`
### Exit fill
The fields in `webhook.exit_fill` are filled when the bot fills a exit order (closes a Trade). Parameters are filled using string.format.
@@ -5,7 +5,7 @@ We **strongly** recommend that Windows users use [Docker](docker_quickstart.md)
If that is not possible, try using the Windows Linux subsystem (WSL) - for which the Ubuntu instructions should work.
Otherwise, please follow the instructions below.
All instructions assume that python 3.10+ is installed and available.
All instructions assume that python 3.11+ is installed and available.
## Clone the git repository
@@ -38,30 +38,6 @@ cd freqtrade
!!! Hint
Using the [Anaconda Distribution](https://www.anaconda.com/distribution/) under Windows can greatly help with installation problems. Check out the [Anaconda installation section](installation.md#installation-with-conda) in the documentation for more information.
### Install ta-lib
Install ta-lib according to the [ta-lib documentation](https://github.com/TA-Lib/ta-lib-python#windows).
As compiling from source on windows has heavy dependencies (requires a partial visual studio installation), Freqtrade provides these dependencies (in the binary wheel format) for the latest 3 Python versions (3.10, 3.11 and 3.12) and for 64bit Windows.
These Wheels are also used by CI running on windows, and are therefore tested together with freqtrade.
Other versions must be downloaded from the above link.
``` powershell
cd \path\freqtrade
python -m venv .venv
.venv\Scripts\activate.ps1
# optionally install ta-lib from wheel
# Eventually adjust the below filename to match the downloaded wheel
pip install --find-links build_helpers\ TA-Lib -U
pip install -r requirements.txt
pip install -e .
freqtrade
```
!!! Note "Use Powershell"
The above installation script assumes you're using powershell on a 64bit windows.
Commands for the legacy CMD windows console may differ.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Matthias
