Modify the original three duplicate functions (_profit_short, _profit_long, _profit), and add _profit_handler and _format_profit_message.
Refactor telegram.py and rpc.py.
Sorry for the duplicate functions yesterday, I was a bit rushed.
Both pytest and ruff have passed.
# Added `/profit_long` and `/profit_short` Commands
Users can now use commands like:
- `/profit_long [<n>]`
- `/profit_short [<n>]`
- `/profit [<n>]`
---
## Key Changes Implemented
### `freqtrade/rpc/telegram.py`:
- The `_profit` command handler has been updated to robustly parse `long` or `short` as optional arguments.
- **Translation:** The `_profit` command handler has been improved to reliably interpret `long` or `short` as optional parameters.
- The determined direction is passed to the RPC layer.
- **Translation:** The direction determined (either `long` or `short`) is passed to the RPC layer.
- The `/help` command documentation is updated.
- **Translation:** The documentation for the `/help` command has been updated accordingly.
---
### `freqtrade/rpc/rpc.py`:
- The `_rpc_trade_statistics` method now accepts a direction parameter.
- **Translation:** The `_rpc_trade_statistics` method has been updated to accept a `direction` parameter.
- The method has been refactored into a main function and a `_process_trade_stats` helper function to reduce complexity and improve readability.
- **Translation:** The method has been refactored into a main function and a helper function, `_process_trade_stats`, to reduce complexity and improve readability.
- The database query filter is dynamically modified to include a condition on `Trade.is_short` when a direction is provided.
- **Translation:** The database query filter dynamically adjusts to include a condition on `Trade.is_short` when a direction is specified.
---
### `tests/rpc/test_rpc_telegram.py`:
- Existing tests for `_profit` have been updated to match the new message format.
- **Translation:** Existing tests for the `_profit` function have been updated to match the new message format.
- New test cases have been added to specifically validate the `long` and `short` filtering functionality.
- **Translation:** New test cases have been added to specifically validate the filtering functionality for `long` and `short` trades.
---
## Testing
- All local `pytest` tests pass successfully.
- **Translation:** All local `pytest` tests have passed successfully.
- All `ruff` linter checks pass.
- **Translation:** All `ruff` code checks have passed.
- As I do not have a full local deployment, I am relying on the CI pipeline for final validation.
- **Translation:** Since I don't have a complete local deployment, I am relying on the CI pipeline for final validation.
---
This time, only a little AI was used :)
Except for the translation.
This commit enhances the /profit Telegram command to allow filtering by trade direction.
- The `_profit` handler in `telegram.py` now parses 'long'/'short' arguments and passes the direction to the RPC layer.
- The `_rpc_trade_statistics` method in `rpc.py` is updated to filter trades based on the provided direction. It has also been refactored for lower complexity.
- The `/help` command documentation is updated to reflect the new functionality.
- Corresponding unit tests in `test_rpc_telegram.py` are updated and extended to cover the new cases.
We'll for now issue a warning about this - and use the "current" tier
This way, gaps in tier data (between maxNotional and the next
minNotional) no longer cause an operational exception.
closes#11923
The bug happened since it just checked the length of the list itself, not what it represents. in this case .*/USDT could be any amount of pairs
when the user sets a max_open_trades of let's say 3 then the pairs only have 3 trade slots of whatever amount of pairs it really has and thereby creating a bottleneck.
This just sets the max_open_trades to -1 without even checking it, letting freqtrade itself handle the amount of trades allowed at a given time.
AI generated strategies and configs cause a lot of noise.
we should be clear that people shall read the documentation first
(this will also allow us to point people at this within the issue).
I got the following error on Pi 2 (using Debian Bookworm armhf arch)
```
...
running bdist_wheel
running build
running build_py
creating build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/lock.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/backend_ctypes.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_imp_emulation.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/model.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/ffiplatform.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/setuptools_ext.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/error.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/vengine_gen.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/api.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/__init__.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/recompiler.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/cffi_opcode.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/pkgconfig.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/verifier.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/cparser.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_shimmed_dist_utils.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/commontypes.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/vengine_cpy.py -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_cffi_include.h -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/parse_c_type.h -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_embedding.h -> build/lib.linux-armv7l-cpython-311/cffi
copying src/cffi/_cffi_errors.h -> build/lib.linux-armv7l-cpython-311/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-armv7l-cpython-311/src/c
arm-linux-gnueabihf-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/freqtrade/.venv/include -I/usr/include/python3.11 -c src/c/_cffi_backend.c -o build/temp.linux-armv7l-cpython-311/src/c/_cffi_backend.o
src/c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
15 | #include <ffi.h>
| ^~~~~~~
compilation terminated.
error: command '/usr/bin/arm-linux-gnueabihf-gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for cffi
Failed to build cffi
ERROR: Failed to build installable wheels for some pyproject.toml based projects (cffi)
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Failed installing dependencies
```
It can be Easily solved by installing libffi-dev before hand
There are tests failing when using delayed and wrap_non_picklable_objects as decorator.
until I'll find a solution to run generate_optimizer standalone for analyze_per_epoch=True
with warnings.catch_warnings():
from optuna.exceptions import ExperimentalWarning
warnings.filterwarnings("ignore", category=FutureWarning)
this should be when importing sampler
This version supports numpy2 - while still remaining on ta-lib C of 0.4
This will avoid huge update problems, as the underlying library doesn't need to be updated
freqtrade/optimize/space/optunaspaces.py:39: error: Argument 1 to "__init__" of "IntDistribution" has incompatible type "int | float"; expected "int" [arg-type]
freqtrade/optimize/space/optunaspaces.py:39: error: Argument 2 to "__init__" of "IntDistribution" has incompatible type "int | float"; expected "int" [arg-type]
remove all references for ExtraTreesRegressor and skopt.space
@@ -64,13 +64,12 @@ Please find the complete documentation on the [freqtrade website](https://www.fr
## Features
- [x]**Based on Python 3.10+**: For botting on any operating system - Windows, macOS and Linux.
- [x]**Based on Python 3.11+**: For botting on any operating system - Windows, macOS and Linux.
- [x]**Persistence**: Persistence is achieved through sqlite.
- [x]**Dry-run**: Run the bot without paying money.
- [x]**Backtesting**: Run a simulation of your buy/sell strategy.
- [x]**Strategy Optimization by machine learning**: Use machine learning to optimize your buy/sell strategy parameters with real exchange data.
- [X]**Adaptive prediction modeling**: Build a smart strategy with FreqAI that self-trains to the market via adaptive machine learning methods. [Learn more](https://www.freqtrade.io/en/stable/freqai/)
- [x]**Edge position sizing** Calculate your win rate, risk reward ratio, the best stoploss and adjust your position size before taking a position for each specific market. [Learn more](https://www.freqtrade.io/en/stable/edge/).
- [x]**Whitelist crypto-currencies**: Select which crypto-currency you want to trade or use dynamic whitelists.
- [x]**Blacklist crypto-currencies**: Select which crypto-currency you want to avoid.
- [x]**Builtin WebUI**: Builtin web UI to manage your bot.
@@ -112,7 +111,6 @@ positional arguments:
backtesting-show Show past Backtest results
backtesting-analysis
Backtest Analysis module.
edge Edge module.
hyperopt Hyperopt module.
hyperopt-list List Hyperopt results
hyperopt-show Show details of Hyperopt results
@@ -148,6 +146,8 @@ Telegram is not mandatory. However, this is a great way to control your bot. Mor
-`/stopentry`: Stop entering new trades.
-`/status <trade_id>|[table]`: Lists all or specific open trades.
-`/profit [<n>]`: Lists cumulative profit from all finished trades, over the last n days.
-`/profit_long [<n>]`: Lists cumulative profit from all finished long trades, over the last n days.
-`/profit_short [<n>]`: Lists cumulative profit from all finished short trades, over the last n days.
-`/forceexit <trade_id>|all`: Instantly exits the given trade (Ignoring `minimum_roi`).
-`/fx <trade_id>|all`: Alias to `/forceexit`
-`/performance`: Show performance of each finished trade grouped by pair
@@ -156,6 +156,7 @@ Telegram is not mandatory. However, this is a great way to control your bot. Mor
-`/help`: Show help message.
-`/version`: Show version.
## Development branches
The project is currently setup in two main branches:
@@ -221,7 +222,7 @@ To run this bot we recommend you a cloud instance with a minimum of:
"description":"Telegram chat or group ID. Recommended to be set via environment variable FREQTRADE__TELEGRAM__CHAT_ID",
"type":"string"
},
"topic_id":{
"description":"Telegram topic ID - only applicable for group chats",
"description":"Telegram topic ID - only applicable for group chats. Recommended to be set via environment variable FREQTRADE__TELEGRAM__TOPIC_ID",
"type":"string"
},
"authorized_users":{
@@ -773,9 +769,11 @@
"type":"object",
"properties":{
"enabled":{
"description":"Enable webhook notifications.",
"type":"boolean"
},
"url":{
"description":"Webhook URL. Recommended to be set via environment variable FREQTRADE__WEBHOOK__URL",
"type":"string"
},
"format":{
@@ -853,6 +851,7 @@
"type":"boolean"
},
"webhook_url":{
"description":"Discord webhook URL. Recommended to be set via environment variable FREQTRADE__DISCORD__WEBHOOK_URL",
"type":"string"
},
"exit_fill":{
@@ -1168,28 +1167,35 @@
"description":"Name of the exchange.",
"type":"string"
},
"enable_ws":{
"description":"Enable WebSocket connections to the exchange.",
"type":"boolean",
"default":true
},
"key":{
"description":"API key for the exchange.",
"description":"API key for the exchange. Recommended to be set via environment variable FREQTRADE__EXCHANGE__KEY",
"type":"string",
"default":""
},
"secret":{
"description":"API secret for the exchange.",
"description":"API secret for the exchange. Recommended to be set via environment variable FREQTRADE__EXCHANGE__SECRET",
"type":"string",
"default":""
},
"password":{
"description":"Password for the exchange, if required.",
"description":"Password for the exchange, if required. Recommended to be set via environment variable FREQTRADE__EXCHANGE__PASSWORD",
"type":"string",
"default":""
},
"uid":{
"description":"User ID for the exchange, if required.",
"description":"User ID for the exchange, if required. Recommended to be set via environment variable FREQTRADE__EXCHANGE__UID",
"type":"string"
},
"account_id":{
"description":"Account ID for the exchange, if required. Recommended to be set via environment variable FREQTRADE__EXCHANGE__ACCOUNT_ID",
"type":"string"
},
"wallet_address":{
"description":"Wallet address for the exchange, if required. Usually used by DEX exchanges. Recommended to be set via environment variable FREQTRADE__EXCHANGE__WALLET_ADDRESS",
"type":"string"
},
"private_key":{
"description":"Private key for the exchange, if required. Usually used by DEX exchanges. Recommended to be set via environment variable FREQTRADE__EXCHANGE__PRIVATE_KEY",
"type":"string"
},
"pair_whitelist":{
@@ -1213,6 +1219,11 @@
"type":"boolean",
"default":false
},
"enable_ws":{
"description":"Enable WebSocket connections to the exchange.",
Possible values are either one of "GP", "RF", "ET", "GBRT" (Details can be found in the [scikit-optimize documentation](https://scikit-optimize.github.io/)), or "an instance of a class that inherits from `RegressorMixin` (from sklearn) and where the `predict` method has an optional `return_std` argument, which returns `std(Y | x)` along with `E[Y | x]`".
Possible values are either one of "NSGAIISampler", "TPESampler", "GPSampler", "CmaEsSampler", "NSGAIIISampler", "QMCSampler" (Details can be found in the [optuna-samplers documentation](https://optuna.readthedocs.io/en/stable/reference/samplers/index.html)), or "an instance of a class that inherits from `optuna.samplers.BaseSampler`".
Some research will be necessary to find additional Regressors.
Example for `ExtraTreesRegressor` ("ET") with additional parameters:
# Corresponds to "ET" - but allows additional parameters.
return ExtraTreesRegressor(n_estimators=100)
```
The `dimensions` parameter is the list of `skopt.space.Dimension` objects corresponding to the parameters to be optimized. It can be used to create isotropic kernels for the `skopt.learning.GaussianProcessRegressor` estimator. Here's an example:
Some research will be necessary to find additional Samplers (from optunahub) for example.
!!! Note
While custom estimators can be provided, it's up to you as User to do research on possible parameters and analyze / understand which ones should be used.
If you're unsure about this, best use one of the Defaults (`"ET"` has proven to be the most versatile) without further parameters.
If you're unsure about this, best use one of the Defaults (`"NSGAIIISampler"` has proven to be the most versatile) without further parameters.
The syslog address can be either a Unix domain socket (socket filename) or a UDP socket specification, consisting of IP address and UDP port, separated by the `:` character.
So, the following are the examples of possible addresses:
* `"address": "/dev/log"` -- log to syslog (rsyslog) using the `/dev/log` socket, suitable for most systems.
@@ -323,20 +322,18 @@ So, the following are the examples of possible addresses:
* `"address": "localhost:514"` -- log to local syslog using UDP socket, if it listens on port 514.
* `"address": "<ip>:514"` -- log to remote syslog at IP address and port 514. This may be used on Windows for remote logging to an external syslog server.
??? Info "Deprecated - configure syslog via command line"
`--logfile syslog:<syslog_address>` -- send log messages to `syslog` service using the `<syslog_address>` as the syslog address.
`--logfile syslog:<syslog_address>` -- send log messages to `syslog` service using the `<syslog_address>` as the syslog address.
The syslog address can be either a Unix domain socket (socket filename) or a UDP socket specification, consisting of IP address and UDP port, separated by the `:` character.
The syslog address can be either a Unix domain socket (socket filename) or a UDP socket specification, consisting of IP address and UDP port, separated by the `:` character.
So, the following are the examples of possible usages:
So, the following are the examples of possible usages:
* `--logfile syslog:/dev/log` -- log to syslog (rsyslog) using the `/dev/log` socket, suitable for most systems.
* `--logfile syslog` -- same as above, the shortcut for `/dev/log`.
* `--logfile syslog:/var/run/syslog` -- log to syslog (rsyslog) using the `/var/run/syslog` socket. Use this on MacOS.
* `--logfile syslog:localhost:514` -- log to local syslog using UDP socket, if it listens on port 514.
* `--logfile syslog:<ip>:514` -- log to remote syslog at IP address and port 514. This may be used on Windows for remote logging to an external syslog server.
* `--logfile syslog:/dev/log` -- log to syslog (rsyslog) using the `/dev/log` socket, suitable for most systems.
* `--logfile syslog` -- same as above, the shortcut for `/dev/log`.
* `--logfile syslog:/var/run/syslog` -- log to syslog (rsyslog) using the `/var/run/syslog` socket. Use this on MacOS.
* `--logfile syslog:localhost:514` -- log to local syslog using UDP socket, if it listens on port 514.
* `--logfile syslog:<ip>:514` -- log to remote syslog at IP address and port 514. This may be used on Windows for remote logging to an external syslog server.
@@ -5,6 +5,8 @@ This page explains how to validate your strategy performance by using Backtestin
Backtesting requires historic data to be available.
To learn how to get data for the pairs and exchange you're interested in, head over to the [Data Downloading](data-download.md) section of the documentation.
Backtesting is also available in [webserver mode](freq-ui.md#backtesting), which allows you to run backtests via the web interface.
## Backtesting command reference
--8<-- "commands/backtesting.md"
@@ -319,6 +321,7 @@ It contains some useful key metrics about performance of your strategy on backte
| SQN | 2.45 |
| Profit factor | 1.11 |
| Expectancy (Ratio) | -0.15 (-0.05) |
| Avg. daily profit | 0.0001 BTC |
| Avg. stake amount | 0.001 BTC |
| Total trade volume | 0.429 BTC |
| | |
@@ -372,9 +375,11 @@ It contains some useful key metrics about performance of your strategy on backte
-`Calmar`: Annualized Calmar ratio.
-`SQN`: System Quality Number (SQN) - by Van Tharp.
-`Profit factor`: profit / loss.
-`Expectancy (Ratio)`: Expectancy ratio, which is the average profit or loss per trade. A negative expectancy ratio means that your strategy is not profitable.
-`Avg. daily profit`: Average profit per day, calculated as `(Total Profit / Backtest Days)`.
-`Avg. stake amount`: Average stake amount, either `stake_amount` or the average when using dynamic stake amount.
-`Total trade volume`: Volume generated on the exchange to reach the above profit.
-`Best Pair` / `Worst Pair`: Best and worst performing pair, and it's corresponding `Tot Profit %`.
-`Best Pair` / `Worst Pair`: Best and worst performing pair (based on absolute profit), and it's corresponding `Tot Profit %`.
-`Best Trade` / `Worst Trade`: Biggest single winning trade and biggest single losing trade.
-`Best day` / `Worst day`: Best and worst day based on daily profit.
-`Days win/draw/lose`: Winning / Losing days (draws are usually days without closed trade).
@@ -435,6 +440,10 @@ To save time, by default backtest will reuse a cached result from within the las
To further analyze your backtest results, freqtrade will export the trades to file by default.
You can then load the trades to perform further analysis as shown in the [data analysis](strategy_analysis_example.md#load-backtest-results-to-pandas-dataframe) backtesting section.
Also, you can use freqtrade in [webserver mode](freq-ui.md#backtesting) to visualize the backtest results in a web interface.
This mode also allows you to load existing backtest results, so you can analyze them without running the backtest again.
For this mode - `--notes "<notes>"` can be used to add notes to the backtest results, which will be shown in the web interface.
### Backtest output file
The output file freqtrade produces is a zip file containing the following files:
@@ -180,7 +180,7 @@ Mandatory parameters are marked as **Required**, which means that they are requi
| `minimal_roi` | **Required.** Set the threshold as ratio the bot will use to exit a trade. [More information below](#understand-minimal_roi). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Dict
| `stoploss` | **Required.** Value as ratio of the stoploss used by the bot. More details in the [stoploss documentation](stoploss.md). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Float (as ratio)
| `trailing_stop` | Enables trailing stoploss (based on `stoploss` in either configuration or strategy file). More details in the [stoploss documentation](stoploss.md#trailing-stop-loss). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Boolean
| `trailing_stop_positive` | Changes stoploss once profit has been reached. More details in the [stoploss documentation](stoploss.md#trailing-stop-loss-custom-positive-loss). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Float
| `trailing_stop_positive` | Changes stoploss once profit has been reached. More details in the [stoploss documentation](stoploss.md#trailing-stop-loss-different-positive-loss). [Strategy Override](#parameters-in-the-strategy). <br> **Datatype:** Float
| `trailing_stop_positive_offset` | Offset on when to apply `trailing_stop_positive`. Percentage value which should be positive. More details in the [stoploss documentation](stoploss.md#trailing-stop-loss-only-once-the-trade-has-reached-a-certain-offset). [Strategy Override](#parameters-in-the-strategy). <br>*Defaults to `0.0` (no offset).* <br> **Datatype:** Float
| `trailing_only_offset_is_reached` | Only apply trailing stoploss when the offset is reached. [stoploss documentation](stoploss.md). [Strategy Override](#parameters-in-the-strategy). <br>*Defaults to `false`.* <br> **Datatype:** Boolean
| `fee` | Fee used during backtesting / dry-runs. Should normally not be configured, which has freqtrade fall back to the exchange default fee. Set as ratio (e.g. 0.001 = 0.1%). Fee is applied twice for each trade, once when buying, once when selling. <br> **Datatype:** Float (as ratio)
@@ -234,7 +234,6 @@ Mandatory parameters are marked as **Required**, which means that they are requi
| `exchange.only_from_ccxt` | Prevent data-download from data.binance.vision. Leaving this as false can greatly speed up downloads, but may be problematic if the site is not available.<br>*Defaults to `false`*<br> **Datatype:** Boolean
| `experimental.block_bad_exchanges` | Block exchanges known to not work with freqtrade. Leave on default unless you want to test if that exchange works now. <br>*Defaults to `true`.* <br> **Datatype:** Boolean
| | **Plugins**
| `edge.*` | Please refer to [edge configuration document](edge.md) for detailed explanation of all possible configuration options.
| `pairlists` | Define one or more pairlists to be used. [More information](plugins.md#pairlists-and-pairlist-handlers). <br>*Defaults to `StaticPairList`.* <br> **Datatype:** List of Dicts
| | **Telegram**
| `telegram.enabled` | Enable the usage of Telegram. <br> **Datatype:** Boolean
@@ -672,7 +671,7 @@ Should you experience problems you suspect are caused by websockets, you can dis
}
```
Should you be required to use a proxy, please refer to the [proxy section](#using-proxy-with-freqtrade) for more information.
Should you be required to use a proxy, please refer to the [proxy section](#using-a-proxy-with-freqtrade) for more information.
!!! Info "Rollout"
We're implementing this out slowly, ensuring stability of your bots.
@@ -304,6 +304,13 @@ The `IProtection` parent class provides a helper method for this in `calculate_l
Most exchanges supported by CCXT should work out of the box.
If you need to implement a specific exchange class, these are found in the `freqtrade/exchange` source folder. You'll also need to add the import to `freqtrade/exchange/__init__.py` to make the loading logic aware of the new exchange.
We recommend looking at existing exchange implementations to get an idea of what might be required.
!!! Warning
Implementing and testing an exchange can be a lot of trial and error, so please bear this in mind.
You should also have some development experience, as this is not a beginner task.
To quickly test the public endpoints of an exchange, add a configuration for your exchange to `tests/exchange_online/conftest.py` and run these tests with `pytest --longrun tests/exchange_online/test_ccxt_compat.py`.
Completing these tests successfully a good basis point (it's a requirement, actually), however these won't guarantee correct exchange functioning, as this only tests public endpoints, but no private endpoint (like generate order or similar).
The `Edge Positioning` module uses probability to calculate your win rate and risk reward ratio. It will use these statistics to control your strategy trade entry points, position size and, stoploss.
!!! Danger "Deprecated functionality"
`Edge positioning` (or short Edge) is currently in maintenance mode only (we keep existing functionality alive) and should be considered as deprecated.
It will currently not receive new features until either someone stepped forward to take up ownership of that module - or we'll decide to remove edge from freqtrade.
!!! Warning
When using `Edge positioning` with a dynamic whitelist (VolumePairList), make sure to also use `AgeFilter` and set it to at least `calculate_since_number_of_days` to avoid problems with missing data.
!!! Note
`Edge Positioning` only considers *its own* buy/sell/stoploss signals. It ignores the stoploss, trailing stoploss, and ROI settings in the strategy configuration file.
`Edge Positioning` improves the performance of some trading strategies and *decreases* the performance of others.
## Introduction
Trading strategies are not perfect. They are frameworks that are susceptible to the market and its indicators. Because the market is not at all predictable, sometimes a strategy will win and sometimes the same strategy will lose.
To obtain an edge in the market, a strategy has to make more money than it loses. Making money in trading is not only about *how often* the strategy makes or loses money.
!!! tip "It doesn't matter how often, but how much!"
A bad strategy might make 1 penny in *ten* transactions but lose 1 dollar in *one* transaction. If one only checks the number of winning trades, it would be misleading to think that the strategy is actually making a profit.
The Edge Positioning module seeks to improve a strategy's winning probability and the money that the strategy will make *on the long run*.
We raise the following question[^1]:
!!! Question "Which trade is a better option?"
a) A trade with 80% of chance of losing 100\$ and 20% chance of winning 200\$<br/>
b) A trade with 100% of chance of losing 30\$
???+ Info "Answer"
The expected value of *a)* is smaller than the expected value of *b)*.<br/>
Hence, *b*) represents a smaller loss in the long run.<br/>
However, the answer is: *it depends*
Another way to look at it is to ask a similar question:
!!! Question "Which trade is a better option?"
a) A trade with 80% of chance of winning 100\$ and 20% chance of losing 200\$<br/>
b) A trade with 100% of chance of winning 30\$
Edge positioning tries to answer the hard questions about risk/reward and position size automatically, seeking to minimizes the chances of losing of a given strategy.
### Trading, winning and losing
Let's call $o$ the return of a single transaction $o$ where $o \in \mathbb{R}$. The collection $O = \{o_1, o_2, ..., o_N\}$ is the set of all returns of transactions made during a trading session. We say that $N$ is the cardinality of $O$, or, in lay terms, it is the number of transactions made in a trading session.
!!! Example
In a session where a strategy made three transactions we can say that $O = \{3.5, -1, 15\}$. That means that $N = 3$ and $o_1 = 3.5$, $o_2 = -1$, $o_3 = 15$.
A winning trade is a trade where a strategy *made* money. Making money means that the strategy closed the position in a value that returned a profit, after all deducted fees. Formally, a winning trade will have a return $o_i > 0$. Similarly, a losing trade will have a return $o_j \leq 0$. With that, we can discover the set of all winning trades, $T_{win}$, as follows:
$$ T_{win} = \{ o \in O | o > 0 \} $$
Similarly, we can discover the set of losing trades $T_{lose}$ as follows:
$$ T_{lose} = \{o \in O | o \leq 0\} $$
!!! Example
In a section where a strategy made four transactions $O = \{3.5, -1, 15, 0\}$:<br>
$T_{win} = \{3.5, 15\}$<br>
$T_{lose} = \{-1, 0\}$<br>
### Win Rate and Lose Rate
The win rate $W$ is the proportion of winning trades with respect to all the trades made by a strategy. We use the following function to compute the win rate:
$$W = \frac{|T_{win}|}{N}$$
Where $W$ is the win rate, $N$ is the number of trades and, $T_{win}$ is the set of all trades where the strategy made money.
Similarly, we can compute the rate of losing trades:
$$
L = \frac{|T_{lose}|}{N}
$$
Where $L$ is the lose rate, $N$ is the amount of trades made and, $T_{lose}$ is the set of all trades where the strategy lost money. Note that the above formula is the same as calculating $L = 1 – W$ or $W = 1 – L$
### Risk Reward Ratio
Risk Reward Ratio ($R$) is a formula used to measure the expected gains of a given investment against the risk of loss. It is basically what you potentially win divided by what you potentially lose. Formally:
$$ R = \frac{\text{potential_profit}}{\text{potential_loss}} $$
???+ Example "Worked example of $R$ calculation"
Let's say that you think that the price of *stonecoin* today is 10.0\$. You believe that, because they will start mining stonecoin, it will go up to 15.0\$ tomorrow. There is the risk that the stone is too hard, and the GPUs can't mine it, so the price might go to 0\$ tomorrow. You are planning to invest 100\$, which will give you 10 shares (100 / 10).
R &= \frac{\text{potential_profit}}{\text{potential_loss}}\\
&= \frac{50}{15}\\
&= 3.33
\end{aligned}$<br>
What it effectively means is that the strategy have the potential to make 3.33\$ for each 1\$ invested.
On a long horizon, that is, on many trades, we can calculate the risk reward by dividing the strategy' average profit on winning trades by the strategy' average loss on losing trades. We can calculate the average profit, $\mu_{win}$, as follows:
By combining the Win Rate $W$ and the Risk Reward ratio $R$ to create an expectancy ratio $E$. A expectance ratio is the expected return of the investment made in a trade. We can compute the value of $E$ as follows:
$$E = R * W - L$$
!!! Example "Calculating $E$"
Let's say that a strategy has a win rate $W = 0.28$ and a risk reward ratio $R = 5$. What this means is that the strategy is expected to make 5 times the investment around on 28% of the trades it makes. Working out the example:<br>
$E = R * W - L = 5 * 0.28 - 0.72 = 0.68$
<br>
The expectancy worked out in the example above means that, on average, this strategy' trades will return 1.68 times the size of its losses. Said another way, the strategy makes 1.68\$ for every 1\$ it loses, on average.
This is important for two reasons: First, it may seem obvious, but you know right away that you have a positive return. Second, you now have a number you can compare to other candidate systems to make decisions about which ones you employ.
It is important to remember that any system with an expectancy greater than 0 is profitable using past data. The key is finding one that will be profitable in the future.
You can also use this value to evaluate the effectiveness of modifications to this system.
!!! Note
It's important to keep in mind that Edge is testing your expectancy using historical data, there's no guarantee that you will have a similar edge in the future. It's still vital to do this testing in order to build confidence in your methodology but be wary of "curve-fitting" your approach to the historical data as things are unlikely to play out the exact same way for future trades.
## How does it work?
Edge combines dynamic stoploss, dynamic positions, and whitelist generation into one isolated module which is then applied to the trading strategy. If enabled in config, Edge will go through historical data with a range of stoplosses in order to find buy and sell/stoploss signals. It then calculates win rate and expectancy over *N* trades for each stoploss. Here is an example:
The goal here is to find the best stoploss for the strategy in order to have the maximum expectancy. In the above example stoploss at $3%$ leads to the maximum expectancy according to historical data.
Edge module then forces stoploss value it evaluated to your strategy dynamically.
### Position size
Edge dictates the amount at stake for each trade to the bot according to the following factors:
- Allowed capital at risk
- Stoploss
Allowed capital at risk is calculated as follows:
```
Allowed capital at risk = (Capital available_percentage) X (Allowed risk per trade)
```
Stoploss is calculated as described above with respect to historical data.
The position size is calculated as follows:
```
Position size = (Allowed capital at risk) / Stoploss
```
Example:
Let's say the stake currency is **ETH** and there is $10$ **ETH** on the wallet. The capital available percentage is $50%$ and the allowed risk per trade is $1\%$. Thus, the available capital for trading is $10 * 0.5 = 5$ **ETH** and the allowed capital at risk would be $5 * 0.01 = 0.05$ **ETH**.
-**Trade 1:** The strategy detects a new buy signal in the **XLM/ETH** market. `Edge Positioning` calculates a stoploss of $2\%$ and a position of $0.05 / 0.02 = 2.5$ **ETH**. The bot takes a position of $2.5$ **ETH** in the **XLM/ETH** market.
-**Trade 2:** The strategy detects a buy signal on the **BTC/ETH** market while **Trade 1** is still open. `Edge Positioning` calculates the stoploss of $4\%$ on this market. Thus, **Trade 2** position size is $0.05 / 0.04 = 1.25$ **ETH**.
!!! Tip "Available Capital $\neq$ Available in wallet"
The available capital for trading didn't change in **Trade 2** even with **Trade 1** still open. The available capital **is not** the free amount in the wallet.
-**Trade 3:** The strategy detects a buy signal in the **ADA/ETH** market. `Edge Positioning` calculates a stoploss of $1\%$ and a position of $0.05 / 0.01 = 5$ **ETH**. Since **Trade 1** has $2.5$ **ETH** blocked and **Trade 2** has $1.25$ **ETH** blocked, there is only $5 - 1.25 - 2.5 = 1.25$ **ETH** available. Hence, the position size of **Trade 3** is $1.25$ **ETH**.
!!! Tip "Available Capital Updates"
The available capital does not change before a position is sold. After a trade is closed the Available Capital goes up if the trade was profitable or goes down if the trade was a loss.
- The strategy detects a sell signal in the **XLM/ETH** market. The bot exits **Trade 1** for a profit of $1$ **ETH**. The total capital in the wallet becomes $11$ **ETH** and the available capital for trading becomes $5.5$ **ETH**.
-**Trade 4** The strategy detects a new buy signal int the **XLM/ETH** market. `Edge Positioning` calculates the stoploss of $2\%$, and the position size of $0.055 / 0.02 = 2.75$ **ETH**.
## Edge command reference
--8<-- "commands/edge.md"
## Configurations
Edge module has following configuration options:
| Parameter | Description |
|------------|-------------|
| `enabled` | If true, then Edge will run periodically. <br>*Defaults to `false`.* <br> **Datatype:** Boolean
| `process_throttle_secs` | How often should Edge run in seconds. <br>*Defaults to `3600` (once per hour).* <br> **Datatype:** Integer
| `calculate_since_number_of_days` | Number of days of data against which Edge calculates Win Rate, Risk Reward and Expectancy. <br> **Note** that it downloads historical data so increasing this number would lead to slowing down the bot. <br>*Defaults to `7`.* <br> **Datatype:** Integer
| `allowed_risk` | Ratio of allowed risk per trade. <br>*Defaults to `0.01` (1%)).* <br> **Datatype:** Float
| `stoploss_range_max` | Maximum stoploss. <br>*Defaults to `-0.10`.* <br> **Datatype:** Float
| `stoploss_range_step` | As an example if this is set to -0.01 then Edge will test the strategy for `[-0.01, -0,02, -0,03 ..., -0.09, -0.10]` ranges. <br> **Note** than having a smaller step means having a bigger range which could lead to slow calculation. <br> If you set this parameter to -0.001, you then slow down the Edge calculation by a factor of 10. <br>*Defaults to `-0.001`.* <br> **Datatype:** Float
| `minimum_winrate` | It filters out pairs which don't have at least minimum_winrate. <br>This comes handy if you want to be conservative and don't comprise win rate in favour of risk reward ratio. <br>*Defaults to `0.60`.* <br> **Datatype:** Float
| `minimum_expectancy` | It filters out pairs which have the expectancy lower than this number. <br>Having an expectancy of 0.20 means if you put 10\$ on a trade you expect a 12\$ return. <br>*Defaults to `0.20`.* <br> **Datatype:** Float
| `min_trade_number` | When calculating *W*, *R* and *E* (expectancy) against historical data, you always want to have a minimum number of trades. The more this number is the more Edge is reliable. <br>Having a win rate of 100% on a single trade doesn't mean anything at all. But having a win rate of 70% over past 100 trades means clearly something. <br>*Defaults to `10` (it is highly recommended not to decrease this number).* <br> **Datatype:** Integer
| `max_trade_duration_minute` | Edge will filter out trades with long duration. If a trade is profitable after 1 month, it is hard to evaluate the strategy based on it. But if most of trades are profitable and they have maximum duration of 30 minutes, then it is clearly a good sign.<br>**NOTICE:** While configuring this value, you should take into consideration your timeframe. As an example filtering out trades having duration less than one day for a strategy which has 4h interval does not make sense. Default value is set assuming your strategy interval is relatively small (1m or 5m, etc.).<br>*Defaults to `1440` (one day).* <br> **Datatype:** Integer
| `remove_pumps` | Edge will remove sudden pumps in a given market while going through historical data. However, given that pumps happen very often in crypto markets, we recommend you keep this off.<br>*Defaults to `false`.* <br> **Datatype:** Boolean
## Running Edge independently
You can run Edge independently in order to see in details the result. Here is an example:
Edge produced the above table by comparing `calculate_since_number_of_days` to `minimum_expectancy` to find `min_trade_number` historical information based on the config file. The timerange Edge uses for its comparisons can be further limited by using the `--timerange` switch.
In live and dry-run modes, after the `process_throttle_secs` has passed, Edge will again process `calculate_since_number_of_days` against `minimum_expectancy` to find `min_trade_number`. If no `min_trade_number` is found, the bot will return "whitelist empty". Depending on the trade strategy being deployed, "whitelist empty" may be return much of the time - or *all* of the time. The use of Edge may also cause trading to occur in bursts, though this is rare.
If you encounter "whitelist empty" a lot, condsider tuning `calculate_since_number_of_days`, `minimum_expectancy` and `min_trade_number` to align to the trading frequency of your strategy.
### Update cached pairs with the latest data
Edge requires historic data the same way as backtesting does.
Please refer to the [Data Downloading](data-download.md) section of the documentation for details.
Doing `--timerange=-20190901` will get all available data until September 1st (excluding September 1st 2019).
The full timerange specification:
* Use tickframes till 2018/01/31: `--timerange=-20180131`
* Use tickframes since 2018/01/31: `--timerange=20180131-`
* Use tickframes since 2018/01/31 till 2018/03/01 : `--timerange=20180131-20180301`
* Use tickframes between POSIX timestamps 1527595200 1527618600: `--timerange=1527595200-1527618600`
[^1]: Question extracted from MIT Opencourseware S096 - Mathematics with applications in Finance: https://ocw.mit.edu/courses/mathematics/18-s096-topics-in-mathematics-with-applications-in-finance-fall-2013/
@@ -339,13 +339,13 @@ This needs to be configured like this:
```json
"exchange": {
"name": "hyperliquid",
"walletAddress": "your_eth_wallet_address",
"walletAddress": "your_eth_wallet_address", // This should NOT be your API Wallet Address!
"privateKey": "your_api_private_key",
// ...
}
```
* walletAddress in hex format: `0x<40 hex characters>` - Can be easily copied from your wallet - and should be your wallet address, not your API Wallet Address.
* walletAddress in hex format: `0x<40 hex characters>` - Can be easily copied from your wallet - and should be your main wallet address, not your API Wallet Address.
* privateKey in hex format: `0x<64 hex characters>` - Use the key the API Wallet shows on creation.
Hyperliquid handles deposits and withdrawals on the Arbitrum One chain, a Layer 2 scaling solution built on top of Ethereum. Hyperliquid uses USDC as quote / collateral. The process of depositing USDC on Hyperliquid requires a couple of steps, see [how to start trading](https://hyperliquid.gitbook.io/hyperliquid-docs/onboarding/how-to-start-trading) for details on what steps are needed.
@@ -363,10 +363,50 @@ Hyperliquid handles deposits and withdrawals on the Arbitrum One chain, a Layer
* Create a different software wallet, only transfer the funds you want to trade with to that wallet, and use that wallet to trade on Hyperliquid.
* If you have funds you don't want to use for trading (after making a profit for example), transfer them back to your hardware wallet.
### Hyperliquid Vault / Subaccount
Hyperliquid allows you to create either a vault or a subaccount.
To use these with Freqtrade, you will need to use the following configuration pattern:
``` json
"exchange": {
"name": "hyperliquid",
"walletAddress": "your_vault_address", // Vault or subaccount address
"privateKey": "your_api_private_key",
"ccxt_config": {
"options": {
"vaultAddress": "your_vault_address" // Optional, only if you want to use a vault or subaccount
}
},
// ...
}
```
Your balance and trades will now be used from your vault / subaccount - and no longer from your main account.
### Historic Hyperliquid data
The Hyperliquid API does not provide historic data beyond the single call to fetch current data, so downloading data is not possible, as the downloaded data would not constitute proper historic data.
## Bitvavo
If your account is required to use an operatorId, you can set it in the configuration file as follows:
``` json
"exchange": {
"name": "bitvavo",
"key": "",
"secret": "",
"ccxt_config": {
"options": {
"operatorId": "123567"
}
},
}
```
Bitvavo expects the `operatorId` to be an integer.
## All exchanges
Should you experience constant errors with Nonce (like `InvalidNonce`), it is best to regenerate the API keys. Resetting Nonce is difficult and it's usually easier to regenerate the API keys.
@@ -102,6 +102,14 @@ You can use "current" market data by using the [dataprovider](strategy-customiza
You can use the `/stopentry` command in Telegram to prevent future trade entry, followed by `/forceexit all` (sell all open trades).
### I sold the bot's capital and now there's errors in the log
Freqtrade assumes that the trades it opens are managed only though the bot.
If you happen to (accidentally) sell the bot's capital, freqtrade will try to recover by trying to re-find on-exchange orders.
This is a best-effort approach, and will not work in all cases, especially when using order types that are not supported by freqtrade (OCO, iceberg, etc.), or when working with older trades (where the exchange no longer provides full order information).
The exact limits will vary between exchanges - with the details usually being documented in the exchange's API documentation.
### I want to run multiple bots on the same machine
Please look at the [advanced setup documentation Page](advanced-setup.md#running-multiple-instances-of-freqtrade).
@@ -151,6 +159,14 @@ This warning can point to one of the below problems:
* Barely traded pair -> Check the pair on the exchange webpage, look at the timeframe your strategy uses. If the pair does not have any volume in some candles (usually visualized with a "volume 0" bar, and a "_" as candle), this pair did not have any trades in this timeframe. These pairs should ideally be avoided, as they can cause problems with order-filling.
* API problem -> API returns wrong data (this only here for completeness, and should not happen with supported exchanges).
### I get the message "Couldn't reuse watch for xxx" in the log
This is an informational message that the bot tried to use candles from the websocket, but the exchange didn't provide the right information.
This can happen if there was an interruption to the websocket connection - or if the pair didn't have any trades happen in the timeframe you are using.
Freqtrade will handle this gracefully by falling back to the REST api.
While this makes the iteration slightly slower (due to the REST Api call) - it will not cause any problems to the bot's operation.
### I'm getting the "Exchange XXX does not support market orders." message and cannot run my strategy
As the message says, your exchange does not support market orders and you have one of the [order types](configuration.md/#understand-order_types) set to "market". Your strategy was probably written with other exchanges in mind and sets "market" orders for "stoploss" orders, which is correct and preferable for most of the exchanges supporting market orders (but not for Gate.io).
@@ -219,10 +235,7 @@ On Windows, the `--logfile` option is also supported by Freqtrade and you can us
First of all, most indicator libraries don't have GPU support - as such, there would be little benefit for indicator calculations.
The GPU improvements would only apply to pandas-native calculations - or ones written by yourself.
For hyperopt, freqtrade is using scikit-optimize, which is built on top of scikit-learn.
Their statement about GPU support is [pretty clear](https://scikit-learn.org/stable/faq.html#will-you-add-gpu-support).
GPU's also are only good at crunching numbers (floating point operations).
GPU's are only good at crunching numbers (floating point operations).
For hyperopt, we need both number-crunching (find next parameters) and running python code (running backtesting).
As such, GPU's are not too well suited for most parts of hyperopt.
@@ -271,20 +284,6 @@ Example: 4% profit 650 times vs 0,3% profit a trade 10000 times in a year. If we
### Edge implements interesting approach for controlling position size, is there any theory behind it?
The Edge module is mostly a result of brainstorming of [@mishaker](https://github.com/mishaker) and [@creslinux](https://github.com/creslinux) freqtrade team members.
You can find further info on expectancy, win rate, risk management and position size in the following sources:
@@ -181,7 +181,7 @@ You can ask for each of the defined features to be included also for informative
In total, the number of features the user of the presented example strategy has created is: length of `include_timeframes`* no. features in `feature_engineering_expand_*()` * length of `include_corr_pairlist` * no. `include_shifted_candles` * length of `indicator_periods_candles`
$= 3 * 3 * 3 * 2 * 2 = 108$.
!!! note "Learn more about creative feature engineering"
!!! note "Learn more about creative feature engineering"
Check out our [medium article](https://emergentmethods.medium.com/freqai-from-price-to-prediction-6fadac18b665) geared toward helping users learn how to creatively engineer features.
### Gain finer control over `feature_engineering_*` functions with `metadata`
@@ -310,7 +310,7 @@ class MyCoolTransform(BaseTransform):
If you have created your own custom `IFreqaiModel` with a custom `train()`/`predict()` function, *and* you still rely on `data_cleaning_train/predict()`, then you will need to migrate to the new pipeline. If your model does *not* rely on `data_cleaning_train/predict()`, then you do not need to worry about this migration.
More details about the migration can be found [here](strategy_migration.md#freqai---new-data-pipeline).
More details about the migration can be found [here](strategy_migration.md#freqai-new-data-pipeline).
If you are using docker, a dedicated tag with FreqAI dependencies is available as `:freqai`. As such - you can replace the image line in your docker compose file with `image: freqtradeorg/freqtrade:stable_freqai`. This image contains the regular FreqAI dependencies. Similar to native installs, Catboost will not be available on ARM based devices. If you would like to use PyTorch or Reinforcement learning, you should use the torch or RL tags, `image: freqtradeorg/freqtrade:stable_freqaitorch`, `image: freqtradeorg/freqtrade:stable_freqairl`.
!!! note "docker-compose-freqai.yml"
We do provide an explicit docker-compose file for this in `docker/docker-compose-freqai.yml` - which can be used via `docker compose -f docker/docker-compose-freqai.yml run ...` - or can be copied to replace the original docker file. This docker-compose file also contains a (disabled) section to enable GPU resources within docker containers. This obviously assumes the system has GPU resources available.
We do provide an explicit docker-compose file for this in `docker/docker-compose-freqai.yml` - which can be used via `docker compose -f docker/docker-compose-freqai.yml run ...` - or can be copied to replace the original docker file. This docker-compose file also contains a (disabled) section to enable GPU resources within docker containers. This obviously assumes the system has GPU resources available.
### FreqAI position in open-source machine learning landscape
This page explains how to tune your strategy by finding the optimal
parameters, a process called hyperparameter optimization. The bot uses algorithms included in the `scikit-optimize` package to accomplish this.
parameters, a process called hyperparameter optimization. The bot uses algorithms included in the `optuna` package to accomplish this.
The search will burn all your CPU cores, make your laptop sound like a fighter jet and still take a long time.
In general, the search for best parameters starts with a few random combinations (see [below](#reproducible-results) for more details) and then uses Bayesian search with a ML regressor algorithm (currently ExtraTreesRegressor) to quickly find a combination of parameters in the search hyperspace that minimizes the value of the [loss function](#loss-functions).
In general, the search for best parameters starts with a few random combinations (see [below](#reproducible-results) for more details) and then uses one of optuna's sampler algorithms (currently NSGAIIISampler) to quickly find a combination of parameters in the search hyperspace that minimizes the value of the [loss function](#loss-functions).
Hyperopt requires historic data to be available, just as backtesting does (hyperopt runs backtesting many times with different parameters).
To learn how to get data for the pairs and exchange you're interested in, head over to the [Data Downloading](data-download.md) section of the documentation.
The `-e` option will set how many evaluations hyperopt will do. Since hyperopt uses Bayesian search, running too many epochs at once may not produce greater results. Experience has shown that best results are usually not improving much after 500-1000 epochs.
The `--early-stop` option will set after how many epochs with no improvements hyperopt will stop. A good value is 20-30% of the total epochs. Any value greater than 0 and lower than 20 it will be replaced by 20. Early stop is by default disabled (`--early-stop=0`)
Doing multiple runs (executions) with a few 1000 epochs and different random state will most likely produce different results.
The `--spaces all` option determines that all possible parameters should be optimized. Possibilities are listed below.
@@ -532,7 +534,7 @@ Legal values are:
* `trailing`: search for the best trailing stop values
* `trades`: search for the best max open trades values
* `protection`: search for the best protection parameters (read the [protections section](#optimizing-protections) on how to properly define these)
* `default`: `all` except `trailing` and `protection`
* `default`: `all` except `trailing`, `trades` and `protection`
* space-separated list of any of the above values for example `--spaces roi stoploss`
The default Hyperopt Search Space, used when no `--space` command line option is specified, does not include the `trailing` hyperspace. We recommend you to run optimization for the `trailing` hyperspace separately, when the best parameters for other hyperspaces were found, validated and pasted into your custom strategy.
Freqtrade allows your strategy to implement different exit logic using signal-based or callback-based functions.
This section aims to compare each different function, helping you to choose the one that best fits your needs.
* **`populate_exit_trend()`** - Vectorized signal-based exit logic using indicators in the main dataframe
✅ **Use** to define exit signals based on indicators or other data that can be calculated in a vectorized manner.
🚫 **Don't use** to customize exit conditions for each individual trade, or if trade data is necessary to make an exit decision.
* **`custom_exit()`** - Custom exit logic that will fully exit a trade immediately, called for every open trade at every bot loop iteration until a trade is closed.
✅ **Use** to specify exit conditions for each individual trade (including any additional adjusted orders using `adjust_trade_position()`), or if trade data is necessary to make an exit decision, e.g. using profit data to exit.
🚫 **Don't use** when you want to exit using vectorised indicator-based data (use a `populate_exit_trend()` signal instead), or as a proxy for `custom_stoploss()`, and be aware that rate-based exits in backtesting can be inaccurate.
* **`custom_stoploss()`** - Custom trailing stoploss, called for every open trade every iteration until a trade is closed. The value returned here is also used for [stoploss on exchange](stoploss.md#stop-loss-on-exchangefreqtrade).
✅ **Use** to customize the stoploss logic to set a dynamic stoploss based on trade data or other conditions.
🚫 **Don't use** to exit a trade immediately based on a specific condition. Use `custom_exit()` for that purpose.
* **`custom_roi()`** - Custom ROI, called for every open trade every iteration until a trade is closed.
✅ **Use** to specify a minimum ROI threshold ("take-profit") to exit a trade at this ROI level at some point within the trade duration, based on profit or other conditions.
🚫 **Don't use** to exit a trade immediately based on a specific condition. Use `custom_exit()`.
🚫 **Don't use** for static ROI. Use `minimal_roi`.
@@ -31,7 +31,6 @@ Freqtrade is a free and open source crypto trading bot written in Python. It is
- Optimize: Find the best parameters for your strategy using hyperoptimization which employs machine learning methods. You can optimize buy, sell, take profit (ROI), stop-loss and trailing stop-loss parameters for your strategy.
- Select markets: Create your static list or use an automatic one based on top traded volumes and/or prices (not available during backtesting). You can also explicitly blacklist markets you don't want to trade.
- Run: Test your strategy with simulated money (Dry-Run mode) or deploy it with real money (Live-Trade mode).
- Run using Edge (optional module): The concept is to find the best historical [trade expectancy](edge.md#expectancy) by markets based on variation of the stop-loss and then allow/reject markets to trade. The sizing of the trade is based on a risk of a percentage of your capital.
- Control/Monitor: Use Telegram or a WebUI (start/stop the bot, show profit/loss, daily summary, current open trades results, etc.).
- Analyze: Further analysis can be performed on either Backtesting data or Freqtrade trading history (SQL database), including automated standard plots, and methods to load the data into [interactive environments](data-analysis.md).
@@ -88,7 +87,7 @@ To run this bot we recommend you a linux cloud instance with a minimum of:
@@ -24,7 +24,7 @@ The easiest way to install and run Freqtrade is to clone the bot Github reposito
The `stable` branch contains the code of the last release (done usually once per month on an approximately one week old snapshot of the `develop` branch to prevent packaging bugs, so potentially it's more stable).
!!! Note
Python3.10 or higher and the corresponding `pip` are assumed to be available. The install-script will warn you and stop if that's not the case. `git` is also needed to clone the Freqtrade repository.
Python3.11 or higher and the corresponding `pip` are assumed to be available. The install-script will warn you and stop if that's not the case. `git` is also needed to clone the Freqtrade repository.
Also, python headers (`python<yourversion>-dev` / `python<yourversion>-devel`) must be available for the installation to complete successfully.
!!! Warning "Up-to-date clock"
@@ -42,7 +42,7 @@ These requirements apply to both [Script Installation](#script-installation) and
This page explains how to validate your strategy in terms of lookahead bias.
This page explains how to validate your strategy in terms of lookahead bias.
Checking lookahead bias is the bane of any strategy since it is sometimes very easy to introduce backtest bias -
but very hard to detect.
Lookahead bias is the bane of any strategy since it is sometimes very easy to introduce this bias, but can be very hard to detect.
Backtesting initializes all timestamps at once and calculates all indicators in the beginning.
This means that if your indicators or entry/exit signals could look into future candles and falsify your backtest.
Backtesting initializes all timestamps (loads the whole dataframe into memory) and calculates all indicators at once.
This means that if your indicators or entry/exit signals look into future candles, this will falsify your backtest.
Lookahead-analysis requires historic data to be available.
The `lookahead-analysis` command requires historic data to be available.
To learn how to get data for the pairs and exchange you're interested in,
head over to the [Data Downloading](data-download.md) section of the documentation.
`lookahead-analysis` also supports freqai strategies.
This command is built upon backtesting since it internally chains backtests and pokes at the strategy to provoke it to show lookahead bias.
This is done by not looking at the strategy itself - but at the results it returned.
The results are things like changed indicator-values and moved entries/exits compared to the full backtest.
This command internally chains backtests and pokes at the strategy to provoke it to show lookahead bias.
This is done by not looking at the strategy code itself, but at changed indicator values and moved entries/exits compared to the full backtest.
You can use commands of [Backtesting](backtesting.md).
It also supports the lookahead-analysis of freqai strategies.
`lookahead-analysis` can use the typical options of [Backtesting](backtesting.md), but forces the following options:
- `--cache` is forced to "none".
- `--max-open-trades` is forced to be at least equal to the number of pairs.
@@ -25,48 +23,83 @@ It also supports the lookahead-analysis of freqai strategies.
- `--stake-amount` is forced to be a static 10000 (10k).
- `--enable-protections` is forced to be off.
Those are set to avoid users accidentally generating false positives.
These are set to avoid users accidentally generating false positives.
## Lookahead-analysis command reference
--8<-- "commands/lookahead-analysis.md"
!!! Note ""
The above Output was reduced to options `lookahead-analysis` adds on top of regular backtesting commands.
### Summary
Checks a given strategy for look ahead bias via lookahead-analysis
Look ahead bias means that the backtest uses data from future candles thereby not making it viable beyond backtesting
and producing false hopes for the one backtesting.
!!! Note
The above output was reduced to options that `lookahead-analysis` adds on top of regular backtesting commands.
### Introduction
Many strategies - without the programmer knowing - have fallen prey to lookahead bias.
Many strategies, without the programmer knowing, have fallen prey to lookahead bias.
This typically makes the strategy backtest look profitable, sometimes to extremes, but this is not realistic as the strategy is "cheating" by looking at data it would not have in dry or live modes.
Any backtest will populate the full dataframe including all timestamps at the beginning.
If the programmer is not careful or oblivious how things work internally
(which sometimes can be really hard to find out) then it will just look into the future making the strategy amazing
but not realistic.
The reason why strategies can "cheat" is because the freqtrade backtesting process populates the full dataframe including all candle timestamps at the outset.
If the programmer is not careful or oblivious how things work internally
(which sometimes can be really hard to find out) then the strategy will look into the future.
This command is made to try to verify the validity in the form of the aforementioned lookahead bias.
This command is made to try to verify the validity in the form of the aforementioned lookahead bias.
### How does the command work?
It will start with a backtest of all pairs to generate a baseline for indicators and entries/exits.
After the backtest ran, it will look if the `minimum-trade-amount` is met
and if not cancel the lookahead-analysis for this strategy.
After this initial backtest runs, it will look if the `minimum-trade-amount` is met and if not cancel the lookahead-analysis for this strategy.
If this happens, use a wider timerange to get more trades for the analysis, or use a timerange where more trades occur.
After setting the baseline it will then do additional runs for every entry and exit separately.
When a verification-backtest is done, it will compare the indicators as the signal (either entry or exit) and report the bias.
After all signals have been verified or falsified a result-table will be generated for the user to see.
After setting the baseline it will then do additional backtest runs for every entry and exit separately.
When these verificationbacktests complete, it will compare the indicators at the signal candles (both entry or exit)
and report the bias.
After all signals have been verified or falsified a result table will be generated for the user to see.
### How to find and remove bias? How can I salvage a biased strategy?
If you found a biased strategy online and want to have the same results, just without bias,
then you will be out of luck most of the time.
Usually the bias in the strategy is THE driving factor for "too good to be true" profits.
Removing conditions or indicators that push the profits up from bias will usually make the strategy significantly worse.
You might be able to salvage it partially if the biased indicators or conditions are not the core of the strategy, or there
are other entry and exit signals that are not biased.
### Examples of lookahead-bias
- `shift(-10)` looks 10 candles into the future.
- Using `iloc[]` in populate_* functions to access a specific row in the dataframe.
- For-loops are prone to introduce lookahead bias if you don't tightly control which numbers are looped through.
- Aggregation functions like `.mean()`, `.min()` and `.max()`, without a rolling window,
will calculate the value over the **whole** dataframe, so the signal candle will "see" a value including future candles.
A non-biased example would be to look back candles using `rolling()` instead:
e.g. `dataframe['volume_mean_12'] = dataframe['volume'].rolling(12).mean()`
- `ta.MACD(dataframe, 12, 26, 1)` will introduce bias with a signalperiod of 1.
### What do the columns in the results table mean?
- `filename`: name of the checked strategy file
- `strategy`: checked strategy class name
- `has_bias`: result of the lookahead-analysis. `No` would be good, `Yes` would be bad.
- `total_signals`: number of checked signals (default is 20)
- `biased_entry_signals`: found bias in that many entries
- `biased_exit_signals`: found bias in that many exits
- `biased_indicators`: shows you the indicators themselves that are defined in populate_indicators
You might get false positives in the `biased_exit_signals` if you have biased entry signals paired with those exits.
However, a biased entry will usually result in a biased exit too,
even if the exit itself does not produce the bias -
especially if your entry and exit conditions use the same biased indicator.
**Address the bias in the entries first, then address the exits.**
### Caveats
- `lookahead-analysis` can only verify / falsify the trades it calculated and verified.
If the strategy has many different signals / signal types, it's up to you to select appropriate parameters to ensure that all signals have triggered at least once. Not triggered signals will not have been verified.
This could lead to a false-negative (the strategy will then be reported as non-biased).
- `lookahead-analysis` has access to everything that backtesting has too.
Please don't provoke any configs like enabling position stacking.
If you decide to do so, then make doubly sure that you won't ever run out of `max_open_trades`amount and neither leftover money in your wallet.
- In the results table, the `biased_indicators` column will falsely flag FreqAI target indicators defined in `set_freqai_targets()` as biased. These are not biased and can safely be ignored.
If the strategy has many different signals / signal types, it's up to you to select appropriate parameters to ensure that all signals have triggered at least once. Signals that are not triggered will not have been verified.
This would lead to a false-negative, i.e. the strategy will be reported as non-biased.
- `lookahead-analysis` has access to the same backtesting options and this can introduce problems.
Please don't use any options like enabling position stacking as this will distort the number of checked signals.
If you decide to do so, then make doubly sure that you won't ever run out of `max_open_trades`slots,
and that you have enough capital in the backtest wallet configuration.
- In the results table, the `biased_indicators` column
will falsely flag FreqAI target indicators defined in `set_freqai_targets()` as biased.
**These are not biased and can safely be ignored.**
@@ -50,6 +50,7 @@ Enable subscribing to an instance by adding the `external_message_consumer` sect
| `ping_timeout` | Ping timeout <br>*Defaults to `10`.*<br> **Datatype:** Integer - in seconds.
| `sleep_time` | Sleep time before retrying to connect.<br>*Defaults to `10`.*<br> **Datatype:** Integer - in seconds.
| `remove_entry_exit_signals` | Remove signal columns from the dataframe (set them to 0) on dataframe receipt.<br>*Defaults to `false`.*<br> **Datatype:** Boolean.
| `initial_candle_limit` | Initial candles to expect from the Producer.<br>*Defaults to `1500`.*<br> **Datatype:** Integer - Number of candles.
| `message_size_limit` | Size limit per message<br>*Defaults to `8`.*<br> **Datatype:** Integer - Megabytes.
Instead of (or as well as) calculating indicators in `populate_indicators()` the follower instance listens on the connection to a producer instance's messages (or multiple producer instances in advanced configurations) and requests the producer's most recently analyzed dataframes for each pair in the active whitelist.
:param trade_id: Deletes the trade with this ID from the database.
edge
Return information about edge.
forcebuy
Buy an asset.
@@ -368,7 +365,6 @@ All endpoints in the below table need to be prefixed with the base URL of the AP
| `/blacklist` | GET | Show the current blacklist.
| `/blacklist` | POST | Adds the specified pair to the blacklist.<br/>*Params:*<br/>- `pair` (`str`)
| `/blacklist` | DELETE | Deletes the specified list of pairs from the blacklist.<br/>*Params:*<br/>- `[pair,pair]` (`list[str]`)
| `/edge` | GET | Show validated pairs by Edge if it is enabled.
| `/pair_candles` | GET | Returns dataframe for a pair / timeframe combination while the bot is running. **Alpha**
| `/pair_candles` | POST | Returns dataframe for a pair / timeframe combination while the bot is running, filtered by a provided list of columns to return. **Alpha**<br/>*Params:*<br/>- `<column_list>` (`list[str]`)
| `/pair_history` | GET | Returns an analyzed dataframe for a given timerange, analyzed by a given strategy. **Alpha**
@@ -256,4 +256,4 @@ The new stoploss value will be applied to open trades (and corresponding log-mes
### Limitations
Stoploss values cannot be changed if `trailing_stop` is enabled and the stoploss has already been adjusted, or if [Edge](edge.md) is enabled (since Edge would recalculate stoploss based on the current market situation).
Stoploss values cannot be changed if `trailing_stop` is enabled and the stoploss has already been adjusted.
A simple callback which is called once when the strategy is loaded.
@@ -121,7 +125,7 @@ Freqtrade will fall back to the `proposed_stake` value should your code raise an
Called for open trade every throttling iteration (roughly every 5 seconds) until a trade is closed.
Allows to define custom exit signals, indicating that specified position should be sold. This is very useful when we need to customize exit conditions for each individual trade, or if you need trade data to make an exit decision.
Allows to define custom exit signals, indicating that specified position should be closed (full exit). This is very useful when we need to customize exit conditions for each individual trade, or if you need trade data to make an exit decision.
For example you could implement a 1:2 risk-reward ROI with `custom_exit()`.
@@ -178,6 +182,8 @@ Returning `None` will be interpreted as "no desire to change", and is the only s
Stoploss on exchange works similar to `trailing_stop`, and the stoploss on exchange is updated as configured in `stoploss_on_exchange_interval` ([More details about stoploss on exchange](stoploss.md#stop-loss-on-exchangefreqtrade)).
If you're on futures markets, please take note of the [stoploss and leverage](stoploss.md#stoploss-and-leverage) section, as the stoploss value returned from `custom_stoploss` is the risk for this trade - not the relative price movement.
!!! Note "Use of dates"
All time-based calculations should be done based on `current_time` - using `datetime.now()` or `datetime.utcnow()` is discouraged, as this will break backtesting support.
@@ -233,7 +239,7 @@ class AwesomeStrategy(IStrategy):
:param **kwargs: Ensure to keep this here so updates to this won't break your strategy.
:return float: New stoploss value, relative to the current_rate
"""
return -0.04
return -0.04 * trade.leverage
```
#### Time based trailing stop
@@ -255,9 +261,9 @@ class AwesomeStrategy(IStrategy):
# Make sure you have the longest interval first - these conditions are evaluated from top to bottom.
if current_time - timedelta(minutes=120) > trade.open_date_utc:
@@ -497,6 +503,135 @@ The helper function `stoploss_from_absolute()` can be used to convert from an ab
---
## Custom ROI
Called for open trade every iteration (roughly every 5 seconds) until a trade is closed.
The usage of the custom ROI method must be enabled by setting `use_custom_roi=True` on the strategy object.
This method allows you to define a custom minimum ROI threshold for exiting a trade, expressed as a ratio (e.g., `0.05` for 5% profit). If both `minimal_roi` and `custom_roi` are defined, the lower of the two thresholds will trigger an exit. For example, if `minimal_roi` is set to `{"0": 0.10}` (10% at 0 minutes) and `custom_roi` returns `0.05`, the trade will exit when the profit reaches 5%. Also, if `custom_roi` returns `0.10` and `minimal_roi` is set to `{"0": 0.05}` (5% at 0 minutes), the trade will be closed when the profit reaches 5%.
The method must return a float representing the new ROI threshold as a ratio, or `None` to fall back to the `minimal_roi` logic. Returning `NaN` or `inf` values is considered invalid and will be treated as `None`, causing the bot to use the `minimal_roi` configuration.
### Custom ROI examples
The following examples illustrate how to use the `custom_roi` function to implement different ROI logics.
#### Custom ROI per side
Use different ROI thresholds depending on the `side`. In this example, 5% for long entries and 2% for short entries.
By default, freqtrade use the orderbook to automatically set an order price([Relevant documentation](configuration.md#prices-used-for-orders)), you also have the option to create custom order prices based on your strategy.
@@ -1107,3 +1242,119 @@ class AwesomeStrategy(IStrategy):
return None
```
## Plot annotations callback
The plot annotations callback is called whenever freqUI requests data to display a chart.
This callback has no meaning in the trade cycle context and is only used for charting purposes.
The strategy can then return a list of `AnnotationType` objects to be displayed on the chart.
Depending on the content returned - the chart can display horizontal areas, vertical areas, or boxes.
The full object looks like this:
``` json
{
"type": "area", // Type of the annotation, currently only "area" is supported
"start": "2024-01-01 15:00:00", // Start date of the area
"end": "2024-01-01 16:00:00", // End date of the area
"y_start": 94000.2, // Price / y axis value
"y_end": 98000, // Price / y axis value
"color": "",
"label": "some label"
}
```
The below example will mark the chart with areas for the hours 8 and 15, with a grey color, highlighting the market open and close hours.
@@ -229,6 +229,7 @@ official commands. You can ask at any moment for help with `/help`.
| `/cancel_open_order <trade_id> | /coo <trade_id>` | Cancel an open order for a trade.
| **Metrics** |
| `/profit [<n>]` | Display a summary of your profit/loss from close trades and some stats about your performance, over the last n days (all trades by default)
| `/profit_[long|short] [<n>]` | Display a summary of your profit/loss from close trades in one direction and some stats about your performance, over the last n days (all trades by default)
| `/performance` | Show performance of each finished trade grouped by pair
| `/balance` | Show bot managed balance per currency
| `/balance full` | Show account balance per currency
@@ -240,7 +241,6 @@ official commands. You can ask at any moment for help with `/help`.
| `/entries` | Shows Wins / losses by Exit reason as well as Avg. holding durations for buys and sells
| `/whitelist [sorted] [baseonly]` | Show the current whitelist. Optionally display in alphabetical order and/or with just the base currency of each pairing.
| `/blacklist [pair]` | Show the current blacklist, or adds a pair to the blacklist.
| `/edge` | Show validated pairs by Edge if it is enabled.
## Telegram commands in action
@@ -310,6 +310,8 @@ current max
### /profit
Also available as `/profit_long` and `/profit_short` to show profit for long or short trades only.
Return a summary of your profit/loss and performance.
> **ROI:** Close trades
@@ -451,21 +453,6 @@ Use `/reload_config` to reset the blacklist.
> Using blacklist `StaticPairList` with 2 pairs
>`DODGE/BTC`, `HOT/BTC`.
### /edge
Shows pairs validated by Edge along with their corresponding win-rate, expectancy and stoploss values.
@@ -134,15 +134,17 @@ Most properties here can be None as they are dependent on the exchange response.
|------------|-------------|-------------|
| `trade` | Trade | Trade object this order is attached to |
| `ft_pair` | string | Pair this order is for |
| `ft_is_open` | boolean | is the order filled? |
| `ft_is_open` | boolean | is the order still open? |
| `order_type` | string | Order type as defined on the exchange - usually market, limit or stoploss |
| `status` | string | Status as defined by ccxt. Usually open, closed, expired or canceled |
| `side` | string | Buy or Sell |
| `status` | string | Status as defined by [ccxt's order structure](https://docs.ccxt.com/#/README?id=order-structure). Usually open, closed, expired, canceled or rejected |
| `side` | string | buy or sell |
| `price` | float | Price the order was placed at |
| `average` | float | Average price the order filled at |
| `amount` | float | Amount in base currency |
| `filled` | float | Filled amount (in base currency) |
| `remaining` | float | Remaining amount |
| `filled` | float | Filled amount (in base currency) (use `safe_filled` instead) |
| `safe_filled` | float | Filled amount (in base currency) - guaranteed to not be None |
| `safe_remaining` | float | Remaining amount - either taken from the exchange or calculated. |
| `cost` | float | Cost of the order - usually average * filled (*Exchange dependent on futures, may contain the cost with or without leverage and may be in contracts.*) |
| `stake_amount` | float | Stake amount used for this order. *Added in 2023.7.* |
| `stake_amount_filled` | float | Filled Stake amount used for this order. *Added in 2024.11.* |
Besides the Live-Trade and Dry-Run run modes, the `backtesting`, `edge` and `hyperopt` optimization subcommands, and the `download-data` subcommand which prepares historical data, the bot contains a number of utility subcommands. They are described in this section.
Besides the Live-Trade and Dry-Run run modes, the `backtesting` and `hyperopt` optimization subcommands, and the `download-data` subcommand which prepares historical data, the bot contains a number of utility subcommands. They are described in this section.
@@ -117,9 +117,9 @@ Different payloads can be configured for different events. Not all fields are ne
## Webhook Message types
### Entry
### Entry / Entry fill
The fields in `webhook.entry` are filled when the bot executes a long/short. Parameters are filled using string.format.
The fields in `webhook.entry` and `webhook.entry_fill` are filled when the bot places a long/short Order to increase a position, or when that order fills respectively. Parameters are filled using string.format.
Possible parameters are:
* `trade_id`
@@ -162,31 +162,9 @@ Possible parameters are:
* `current_rate`
* `enter_tag`
### Entry fill
### Exit / Exit fill
The fields in `webhook.entry_fill` are filled when the bot filled a long/short order. Parameters are filled using string.format.
Possible parameters are:
* `trade_id`
* `exchange`
* `pair`
* `direction`
* `leverage`
* `open_rate`
* `amount`
* `open_date`
* `stake_amount`
* `stake_currency`
* `base_currency`
* `quote_currency`
* `fiat_currency`
* `order_type`
* `current_rate`
* `enter_tag`
### Exit
The fields in `webhook.exit` are filled when the bot exits a trade. Parameters are filled using string.format.
The fields in `webhook.exit` and `webhook.exit_fill` are filled when the bot places an exit order, or when that exit order fills respectively. Parameters are filled using string.format.
Possible parameters are:
* `trade_id`
@@ -195,34 +173,9 @@ Possible parameters are:
* `direction`
* `leverage`
* `gain`
* `limit`
* `amount`
* `open_rate`
* `profit_amount`
* `profit_ratio`
* `stake_currency`
* `base_currency`
* `quote_currency`
* `fiat_currency`
* `exit_reason`
* `order_type`
* `open_date`
* `close_date`
### Exit fill
The fields in `webhook.exit_fill` are filled when the bot fills a exit order (closes a Trade). Parameters are filled using string.format.
@@ -5,7 +5,7 @@ We **strongly** recommend that Windows users use [Docker](docker_quickstart.md)
If that is not possible, try using the Windows Linux subsystem (WSL) - for which the Ubuntu instructions should work.
Otherwise, please follow the instructions below.
All instructions assume that python 3.10+ is installed and available.
All instructions assume that python 3.11+ is installed and available.
## Clone the git repository
@@ -42,7 +42,7 @@ cd freqtrade
Install ta-lib according to the [ta-lib documentation](https://github.com/TA-Lib/ta-lib-python#windows).
As compiling from source on windows has heavy dependencies (requires a partial visual studio installation), Freqtrade provides these dependencies (in the binary wheel format) for the latest 3 Python versions (3.10, 3.11 and 3.12) and for 64bit Windows.
As compiling from source on windows has heavy dependencies (requires a partial visual studio installation), Freqtrade provides these dependencies (in the binary wheel format) for the latest 3 Python versions (3.11, 3.12 and 3.13) and for 64bit Windows.
These Wheels are also used by CI running on windows, and are therefore tested together with freqtrade.
Other versions must be downloaded from the above link.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.