mohammadian7/freqtrade
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'freqtrade:develop' into develop

Browse Source
This commit is contained in:
Richard Jozsa
2023-05-22 08:36:25 +00:00
committed by GitHub
parent d26aa231fc 6d9b8a4a99
commit 39f4fb8797
24 changed files with 460 additions and 362 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-compose.yml
+10 -1
View File
@@ -6,6 +6,15 @@ services:
# image: freqtradeorg/freqtrade:develop # image: freqtradeorg/freqtrade:develop
# Use plotting image # Use plotting image
# image: freqtradeorg/freqtrade:develop_plot # image: freqtradeorg/freqtrade:develop_plot
# # Enable GPU Image and GPU Resources (only relevant for freqAI)
# # Make sure to uncomment the whole deploy section
# deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: 1
# capabilities: [gpu]
# Build step - only needed when additional dependencies are needed # Build step - only needed when additional dependencies are needed
# build: # build:
# context: . # context: .
@@ -16,7 +25,7 @@ services:
- "./user_data:/freqtrade/user_data" - "./user_data:/freqtrade/user_data"
# Expose api on port 8080 (localhost only) # Expose api on port 8080 (localhost only)
# Please read the https://www.freqtrade.io/en/stable/rest-api/ documentation # Please read the https://www.freqtrade.io/en/stable/rest-api/ documentation
# before enabling this. # for more information.
ports: ports:
- "127.0.0.1:8080:8080" - "127.0.0.1:8080:8080"
# Default command used when running `docker compose up` # Default command used when running `docker compose up`
docker/docker-compose-freqai.yml
+36
View File
@@ -0,0 +1,36 @@
---
version: '3'
services:
freqtrade:
image: freqtradeorg/freqtrade:stable_freqaitorch
# # Enable GPU Image and GPU Resources
# # Make sure to uncomment the whole deploy section
# deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: 1
# capabilities: [gpu]
# Build step - only needed when additional dependencies are needed
# build:
# context: .
# dockerfile: "./docker/Dockerfile.custom"
restart: unless-stopped
container_name: freqtrade
volumes:
- "./user_data:/freqtrade/user_data"
# Expose api on port 8080 (localhost only)
# Please read the https://www.freqtrade.io/en/stable/rest-api/ documentation
# for more information.
ports:
- "127.0.0.1:8080:8080"
# Default command used when running `docker compose up`
command: >
trade
--logfile /freqtrade/user_data/logs/freqtrade.log
--db-url sqlite:////freqtrade/user_data/tradesv3.sqlite
--config /freqtrade/user_data/config.json
--freqai-model XGBoostClassifier
--strategy SampleStrategy
docs/freqai-configuration.md
+3 -1
View File
@@ -248,9 +248,11 @@ The easiest way to quickly run a pytorch model is with the following command (fo
freqtrade trade --config config_examples/config_freqai.example.json --strategy FreqaiExampleStrategy --freqaimodel PyTorchMLPRegressor --strategy-path freqtrade/templates freqtrade trade --config config_examples/config_freqai.example.json --strategy FreqaiExampleStrategy --freqaimodel PyTorchMLPRegressor --strategy-path freqtrade/templates
``` ```
!!! note "Installation/docker" !!! Note "Installation/docker"
The PyTorch module requires large packages such as `torch`, which should be explicitly requested during `./setup.sh -i` by answering "y" to the question "Do you also want dependencies for freqai-rl or PyTorch (~700mb additional space required) [y/N]?". The PyTorch module requires large packages such as `torch`, which should be explicitly requested during `./setup.sh -i` by answering "y" to the question "Do you also want dependencies for freqai-rl or PyTorch (~700mb additional space required) [y/N]?".
Users who prefer docker should ensure they use the docker image appended with `_freqaitorch`. Users who prefer docker should ensure they use the docker image appended with `_freqaitorch`.
We do provide an explicit docker-compose file for this in `docker/docker-compose-freqai.yml` - which can be used via `docker compose -f docker/docker-compose-freqai.yml run ...` - or can be copied to replace the original docker file.
This docker-compose file also contains a (disabled) section to enable GPU resources within docker containers. This obviously assumes the system has GPU resources available.
### Structure ### Structure
docs/freqai-reinforcement-learning.md
+89 -91
View File
@@ -145,94 +145,94 @@ As you begin to modify the strategy and the prediction model, you will quickly r
The best reward functions are ones that are continuously differentiable, and well scaled. In other words, adding a single large negative penalty to a rare event is not a good idea, and the neural net will not be able to learn that function. Instead, it is better to add a small negative penalty to a common event. This will help the agent learn faster. Not only this, but you can help improve the continuity of your rewards/penalties by having them scale with severity according to some linear/exponential functions. In other words, you'd slowly scale the penalty as the duration of the trade increases. This is better than a single large penalty occuring at a single point in time. The best reward functions are ones that are continuously differentiable, and well scaled. In other words, adding a single large negative penalty to a rare event is not a good idea, and the neural net will not be able to learn that function. Instead, it is better to add a small negative penalty to a common event. This will help the agent learn faster. Not only this, but you can help improve the continuity of your rewards/penalties by having them scale with severity according to some linear/exponential functions. In other words, you'd slowly scale the penalty as the duration of the trade increases. This is better than a single large penalty occuring at a single point in time.
```python ```python
from freqtrade.freqai.prediction_models.ReinforcementLearner import ReinforcementLearner from freqtrade.freqai.prediction_models.ReinforcementLearner import ReinforcementLearner
from freqtrade.freqai.RL.Base5ActionRLEnv import Actions, Base5ActionRLEnv, Positions from freqtrade.freqai.RL.Base5ActionRLEnv import Actions, Base5ActionRLEnv, Positions
class MyCoolRLModel(ReinforcementLearner): class MyCoolRLModel(ReinforcementLearner):
"""
User created RL prediction model.
Save this file to `freqtrade/user_data/freqaimodels`
then use it with:
freqtrade trade --freqaimodel MyCoolRLModel --config config.json --strategy SomeCoolStrat
Here the users can override any of the functions
available in the `IFreqaiModel` inheritance tree. Most importantly for RL, this
is where the user overrides `MyRLEnv` (see below), to define custom
`calculate_reward()` function, or to override any other parts of the environment.
This class also allows users to override any other part of the IFreqaiModel tree.
For example, the user can override `def fit()` or `def train()` or `def predict()`
to take fine-tuned control over these processes.
Another common override may be `def data_cleaning_predict()` where the user can
take fine-tuned control over the data handling pipeline.
"""
class MyRLEnv(Base5ActionRLEnv):
""" """
User created RL prediction model. User made custom environment. This class inherits from BaseEnvironment and gym.env.
Users can override any functions from those parent classes. Here is an example
of a user customized `calculate_reward()` function.
Save this file to `freqtrade/user_data/freqaimodels` Warning!
This is function is a showcase of functionality designed to show as many possible
then use it with: environment control features as possible. It is also designed to run quickly
on small computers. This is a benchmark, it is *not* for live production.
freqtrade trade --freqaimodel MyCoolRLModel --config config.json --strategy SomeCoolStrat
Here the users can override any of the functions
available in the `IFreqaiModel` inheritance tree. Most importantly for RL, this
is where the user overrides `MyRLEnv` (see below), to define custom
`calculate_reward()` function, or to override any other parts of the environment.
This class also allows users to override any other part of the IFreqaiModel tree.
For example, the user can override `def fit()` or `def train()` or `def predict()`
to take fine-tuned control over these processes.
Another common override may be `def data_cleaning_predict()` where the user can
take fine-tuned control over the data handling pipeline.
""" """
class MyRLEnv(Base5ActionRLEnv): def calculate_reward(self, action: int) -> float:
""" # first, penalize if the action is not valid
User made custom environment. This class inherits from BaseEnvironment and gym.env. if not self._is_valid(action):
Users can override any functions from those parent classes. Here is an example return -2
of a user customized `calculate_reward()` function. pnl = self.get_unrealized_profit()
Warning! factor = 100
This is function is a showcase of functionality designed to show as many possible
environment control features as possible. It is also designed to run quickly
on small computers. This is a benchmark, it is *not* for live production.
"""
def calculate_reward(self, action: int) -> float:
# first, penalize if the action is not valid
if not self._is_valid(action):
return -2
pnl = self.get_unrealized_profit()
factor = 100 pair = self.pair.replace(':', '')
pair = self.pair.replace(':', '') # you can use feature values from dataframe
# Assumes the shifted RSI indicator has been generated in the strategy.
rsi_now = self.raw_features[f"%-rsi-period_10_shift-1_{pair}_"
f"{self.config['timeframe']}"].iloc[self._current_tick]
# you can use feature values from dataframe # reward agent for entering trades
# Assumes the shifted RSI indicator has been generated in the strategy. if (action in (Actions.Long_enter.value, Actions.Short_enter.value)
rsi_now = self.raw_features[f"%-rsi-period_10_shift-1_{pair}_" and self._position == Positions.Neutral):
f"{self.config['timeframe']}"].iloc[self._current_tick] if rsi_now < 40:
factor = 40 / rsi_now
else:
factor = 1
return 25 * factor
# reward agent for entering trades # discourage agent from not entering trades
if (action in (Actions.Long_enter.value, Actions.Short_enter.value) if action == Actions.Neutral.value and self._position == Positions.Neutral:
and self._position == Positions.Neutral): return -1
if rsi_now < 40: max_trade_duration = self.rl_config.get('max_trade_duration_candles', 300)
factor = 40 / rsi_now trade_duration = self._current_tick - self._last_trade_tick
else: if trade_duration <= max_trade_duration:
factor = 1 factor *= 1.5
return 25 * factor elif trade_duration > max_trade_duration:
factor *= 0.5
# discourage agent from not entering trades # discourage sitting in position
if action == Actions.Neutral.value and self._position == Positions.Neutral: if self._position in (Positions.Short, Positions.Long) and \
return -1 action == Actions.Neutral.value:
max_trade_duration = self.rl_config.get('max_trade_duration_candles', 300) return -1 * trade_duration / max_trade_duration
trade_duration = self._current_tick - self._last_trade_tick # close long
if trade_duration <= max_trade_duration: if action == Actions.Long_exit.value and self._position == Positions.Long:
factor *= 1.5 if pnl > self.profit_aim * self.rr:
elif trade_duration > max_trade_duration: factor *= self.rl_config['model_reward_parameters'].get('win_reward_factor', 2)
factor *= 0.5 return float(pnl * factor)
# discourage sitting in position # close short
if self._position in (Positions.Short, Positions.Long) and \ if action == Actions.Short_exit.value and self._position == Positions.Short:
action == Actions.Neutral.value: if pnl > self.profit_aim * self.rr:
return -1 * trade_duration / max_trade_duration factor *= self.rl_config['model_reward_parameters'].get('win_reward_factor', 2)
# close long return float(pnl * factor)
if action == Actions.Long_exit.value and self._position == Positions.Long: return 0.
if pnl > self.profit_aim * self.rr:
factor *= self.rl_config['model_reward_parameters'].get('win_reward_factor', 2)
return float(pnl * factor)
# close short
if action == Actions.Short_exit.value and self._position == Positions.Short:
if pnl > self.profit_aim * self.rr:
factor *= self.rl_config['model_reward_parameters'].get('win_reward_factor', 2)
return float(pnl * factor)
return 0.
``` ```
### Using Tensorboard ## Using Tensorboard
Reinforcement Learning models benefit from tracking training metrics. FreqAI has integrated Tensorboard to allow users to track training and evaluation performance across all coins and across all retrainings. Tensorboard is activated via the following command: Reinforcement Learning models benefit from tracking training metrics. FreqAI has integrated Tensorboard to allow users to track training and evaluation performance across all coins and across all retrainings. Tensorboard is activated via the following command:
@@ -245,32 +245,30 @@ where `unique-id` is the `identifier` set in the `freqai` configuration file. Th
![tensorboard](assets/tensorboard.jpg) ![tensorboard](assets/tensorboard.jpg)
## Custom logging
### Custom logging
FreqAI also provides a built in episodic summary logger called `self.tensorboard_log` for adding custom information to the Tensorboard log. By default, this function is already called once per step inside the environment to record the agent actions. All values accumulated for all steps in a single episode are reported at the conclusion of each episode, followed by a full reset of all metrics to 0 in preparation for the subsequent episode. FreqAI also provides a built in episodic summary logger called `self.tensorboard_log` for adding custom information to the Tensorboard log. By default, this function is already called once per step inside the environment to record the agent actions. All values accumulated for all steps in a single episode are reported at the conclusion of each episode, followed by a full reset of all metrics to 0 in preparation for the subsequent episode.
`self.tensorboard_log` can also be used anywhere inside the environment, for example, it can be added to the `calculate_reward` function to collect more detailed information about how often various parts of the reward were called: `self.tensorboard_log` can also be used anywhere inside the environment, for example, it can be added to the `calculate_reward` function to collect more detailed information about how often various parts of the reward were called:
```py ```python
class MyRLEnv(Base5ActionRLEnv): class MyRLEnv(Base5ActionRLEnv):
""" """
User made custom environment. This class inherits from BaseEnvironment and gym.env. User made custom environment. This class inherits from BaseEnvironment and gym.env.
Users can override any functions from those parent classes. Here is an example Users can override any functions from those parent classes. Here is an example
of a user customized `calculate_reward()` function. of a user customized `calculate_reward()` function.
""" """
def calculate_reward(self, action: int) -> float: def calculate_reward(self, action: int) -> float:
if not self._is_valid(action): if not self._is_valid(action):
self.tensorboard_log("invalid") self.tensorboard_log("invalid")
return -2 return -2
``` ```
!!! Note !!! Note
The `self.tensorboard_log()` function is designed for tracking incremented objects only i.e. events, actions inside the training environment. If the event of interest is a float, the float can be passed as the second argument e.g. `self.tensorboard_log("float_metric1", 0.23)`. In this case the metric values are not incremented. The `self.tensorboard_log()` function is designed for tracking incremented objects only i.e. events, actions inside the training environment. If the event of interest is a float, the float can be passed as the second argument e.g. `self.tensorboard_log("float_metric1", 0.23)`. In this case the metric values are not incremented.
### Choosing a base environment ## Choosing a base environment
FreqAI provides three base environments, `Base3ActionRLEnvironment`, `Base4ActionEnvironment` and `Base5ActionEnvironment`. As the names imply, the environments are customized for agents that can select from 3, 4 or 5 actions. The `Base3ActionEnvironment` is the simplest, the agent can select from hold, long, or short. This environment can also be used for long-only bots (it automatically follows the `can_short` flag from the strategy), where long is the enter condition and short is the exit condition. Meanwhile, in the `Base4ActionEnvironment`, the agent can enter long, enter short, hold neutral, or exit position. Finally, in the `Base5ActionEnvironment`, the agent has the same actions as Base4, but instead of a single exit action, it separates exit long and exit short. The main changes stemming from the environment selection include: FreqAI provides three base environments, `Base3ActionRLEnvironment`, `Base4ActionEnvironment` and `Base5ActionEnvironment`. As the names imply, the environments are customized for agents that can select from 3, 4 or 5 actions. The `Base3ActionEnvironment` is the simplest, the agent can select from hold, long, or short. This environment can also be used for long-only bots (it automatically follows the `can_short` flag from the strategy), where long is the enter condition and short is the exit condition. Meanwhile, in the `Base4ActionEnvironment`, the agent can enter long, enter short, hold neutral, or exit position. Finally, in the `Base5ActionEnvironment`, the agent has the same actions as Base4, but instead of a single exit action, it separates exit long and exit short. The main changes stemming from the environment selection include:
docs/freqai.md
+3
View File
@@ -78,6 +78,9 @@ pip install -r requirements-freqai.txt
If you are using docker, a dedicated tag with FreqAI dependencies is available as `:freqai`. As such - you can replace the image line in your docker compose file with `image: freqtradeorg/freqtrade:develop_freqai`. This image contains the regular FreqAI dependencies. Similar to native installs, Catboost will not be available on ARM based devices. If you are using docker, a dedicated tag with FreqAI dependencies is available as `:freqai`. As such - you can replace the image line in your docker compose file with `image: freqtradeorg/freqtrade:develop_freqai`. This image contains the regular FreqAI dependencies. Similar to native installs, Catboost will not be available on ARM based devices.
!!! note "docker-compose-freqai.yml"
We do provide an explicit docker-compose file for this in `docker/docker-compose-freqai.yml` - which can be used via `docker compose -f docker/docker-compose-freqai.yml run ...` - or can be copied to replace the original docker file. This docker-compose file also contains a (disabled) section to enable GPU resources within docker containers. This obviously assumes the system has GPU resources available.
### FreqAI position in open-source machine learning landscape ### FreqAI position in open-source machine learning landscape
Forecasting chaotic time-series based systems, such as equity/cryptocurrency markets, requires a broad set of tools geared toward testing a wide range of hypotheses. Fortunately, a recent maturation of robust machine learning libraries (e.g. `scikit-learn`) has opened up a wide range of research possibilities. Scientists from a diverse range of fields can now easily prototype their studies on an abundance of established machine learning algorithms. Similarly, these user-friendly libraries enable "citzen scientists" to use their basic Python skills for data exploration. However, leveraging these machine learning libraries on historical and live chaotic data sources can be logistically difficult and expensive. Additionally, robust data collection, storage, and handling presents a disparate challenge. [`FreqAI`](#freqai) aims to provide a generalized and extensible open-sourced framework geared toward live deployments of adaptive modeling for market forecasting. The `FreqAI` framework is effectively a sandbox for the rich world of open-source machine learning libraries. Inside the `FreqAI` sandbox, users find they can combine a wide variety of third-party libraries to test creative hypotheses on a free live 24/7 chaotic data source - cryptocurrency exchange data. Forecasting chaotic time-series based systems, such as equity/cryptocurrency markets, requires a broad set of tools geared toward testing a wide range of hypotheses. Fortunately, a recent maturation of robust machine learning libraries (e.g. `scikit-learn`) has opened up a wide range of research possibilities. Scientists from a diverse range of fields can now easily prototype their studies on an abundance of established machine learning algorithms. Similarly, these user-friendly libraries enable "citzen scientists" to use their basic Python skills for data exploration. However, leveraging these machine learning libraries on historical and live chaotic data sources can be logistically difficult and expensive. Additionally, robust data collection, storage, and handling presents a disparate challenge. [`FreqAI`](#freqai) aims to provide a generalized and extensible open-sourced framework geared toward live deployments of adaptive modeling for market forecasting. The `FreqAI` framework is effectively a sandbox for the rich world of open-source machine learning libraries. Inside the `FreqAI` sandbox, users find they can combine a wide variety of third-party libraries to test creative hypotheses on a free live 24/7 chaotic data source - cryptocurrency exchange data.
docs/requirements-docs.txt
+1 -1
View File
@@ -1,6 +1,6 @@
markdown==3.3.7 markdown==3.3.7
mkdocs==1.4.3 mkdocs==1.4.3
mkdocs-material==9.1.12 mkdocs-material==9.1.14
mdx_truly_sane_lists==1.3 mdx_truly_sane_lists==1.3
pymdown-extensions==10.0.1 pymdown-extensions==10.0.1
jinja2==3.1.2 jinja2==3.1.2
freqtrade/constants.py
+2
View File
@@ -690,4 +690,6 @@ BidAsk = Literal['bid', 'ask']
OBLiteral = Literal['asks', 'bids'] OBLiteral = Literal['asks', 'bids']
Config = Dict[str, Any] Config = Dict[str, Any]
# Exchange part of the configuration.
ExchangeConfig = Dict[str, Any]
IntOrInf = float IntOrInf = float
freqtrade/exchange/__init__.py
+1 -1
View File
@@ -1,6 +1,6 @@
# flake8: noqa: F401 # flake8: noqa: F401
# isort: off # isort: off
from freqtrade.exchange.common import remove_credentials, MAP_EXCHANGE_CHILDCLASS from freqtrade.exchange.common import remove_exchange_credentials, MAP_EXCHANGE_CHILDCLASS
from freqtrade.exchange.exchange import Exchange from freqtrade.exchange.exchange import Exchange
# isort: on # isort: on
from freqtrade.exchange.binance import Binance from freqtrade.exchange.binance import Binance
freqtrade/exchange/common.py
+8 -8
View File
@@ -4,7 +4,7 @@ import time
from functools import wraps from functools import wraps
from typing import Any, Callable, Optional, TypeVar, cast, overload from typing import Any, Callable, Optional, TypeVar, cast, overload
from freqtrade.constants import Config from freqtrade.constants import ExchangeConfig
from freqtrade.exceptions import DDosProtection, RetryableOrderError, TemporaryError from freqtrade.exceptions import DDosProtection, RetryableOrderError, TemporaryError
from freqtrade.mixins import LoggingMixin from freqtrade.mixins import LoggingMixin
@@ -89,18 +89,18 @@ EXCHANGE_HAS_OPTIONAL = [
] ]
def remove_credentials(config: Config) -> None: def remove_exchange_credentials(exchange_config: ExchangeConfig, dry_run: bool) -> None:
""" """
Removes exchange keys from the configuration and specifies dry-run Removes exchange keys from the configuration and specifies dry-run
Used for backtesting / hyperopt / edge and utils. Used for backtesting / hyperopt / edge and utils.
Modifies the input dict! Modifies the input dict!
""" """
if config.get('dry_run', False): if dry_run:
config['exchange']['key'] = '' exchange_config['key'] = ''
config['exchange']['apiKey'] = '' exchange_config['apiKey'] = ''
config['exchange']['secret'] = '' exchange_config['secret'] = ''
config['exchange']['password'] = '' exchange_config['password'] = ''
config['exchange']['uid'] = '' exchange_config['uid'] = ''
def calculate_backoff(retrycount, max_retries): def calculate_backoff(retrycount, max_retries):
freqtrade/exchange/exchange.py
+18 -18
View File
@@ -20,16 +20,16 @@ from dateutil import parser
from pandas import DataFrame, concat from pandas import DataFrame, concat
from freqtrade.constants import (DEFAULT_AMOUNT_RESERVE_PERCENT, NON_OPEN_EXCHANGE_STATES, BidAsk, from freqtrade.constants import (DEFAULT_AMOUNT_RESERVE_PERCENT, NON_OPEN_EXCHANGE_STATES, BidAsk,
BuySell, Config, EntryExit, ListPairsWithTimeframes, MakerTaker, BuySell, Config, EntryExit, ExchangeConfig,
OBLiteral, PairWithTimeframe) ListPairsWithTimeframes, MakerTaker, OBLiteral, PairWithTimeframe)
from freqtrade.data.converter import clean_ohlcv_dataframe, ohlcv_to_dataframe, trades_dict_to_list from freqtrade.data.converter import clean_ohlcv_dataframe, ohlcv_to_dataframe, trades_dict_to_list
from freqtrade.enums import OPTIMIZE_MODES, CandleType, MarginMode, TradingMode from freqtrade.enums import OPTIMIZE_MODES, CandleType, MarginMode, TradingMode
from freqtrade.enums.pricetype import PriceType from freqtrade.enums.pricetype import PriceType
from freqtrade.exceptions import (DDosProtection, ExchangeError, InsufficientFundsError, from freqtrade.exceptions import (DDosProtection, ExchangeError, InsufficientFundsError,
InvalidOrderException, OperationalException, PricingError, InvalidOrderException, OperationalException, PricingError,
RetryableOrderError, TemporaryError) RetryableOrderError, TemporaryError)
from freqtrade.exchange.common import (API_FETCH_ORDER_RETRY_COUNT, remove_credentials, retrier, from freqtrade.exchange.common import (API_FETCH_ORDER_RETRY_COUNT, remove_exchange_credentials,
retrier_async) retrier, retrier_async)
from freqtrade.exchange.exchange_utils import (ROUND, ROUND_DOWN, ROUND_UP, CcxtModuleType, from freqtrade.exchange.exchange_utils import (ROUND, ROUND_DOWN, ROUND_UP, CcxtModuleType,
amount_to_contract_precision, amount_to_contracts, amount_to_contract_precision, amount_to_contracts,
amount_to_precision, contracts_to_amount, amount_to_precision, contracts_to_amount,
@@ -92,8 +92,8 @@ class Exchange:
# TradingMode.SPOT always supported and not required in this list # TradingMode.SPOT always supported and not required in this list
] ]
def __init__(self, config: Config, *, validate: bool = True, def __init__(self, config: Config, *, exchange_config: Optional[ExchangeConfig] = None,
load_leverage_tiers: bool = False) -> None: validate: bool = True, load_leverage_tiers: bool = False) -> None:
""" """
Initializes this module with the given config, Initializes this module with the given config,
it does basic validation whether the specified exchange and pairs are valid. it does basic validation whether the specified exchange and pairs are valid.
@@ -131,13 +131,13 @@ class Exchange:
# Holds all open sell orders for dry_run # Holds all open sell orders for dry_run
self._dry_run_open_orders: Dict[str, Any] = {} self._dry_run_open_orders: Dict[str, Any] = {}
remove_credentials(config)
if config['dry_run']: if config['dry_run']:
logger.info('Instance is running with dry_run enabled') logger.info('Instance is running with dry_run enabled')
logger.info(f"Using CCXT {ccxt.__version__}") logger.info(f"Using CCXT {ccxt.__version__}")
exchange_config = config['exchange'] exchange_conf: Dict[str, Any] = exchange_config if exchange_config else config['exchange']
self.log_responses = exchange_config.get('log_responses', False) remove_exchange_credentials(exchange_conf, config.get('dry_run', False))
self.log_responses = exchange_conf.get('log_responses', False)
# Leverage properties # Leverage properties
self.trading_mode: TradingMode = config.get('trading_mode', TradingMode.SPOT) self.trading_mode: TradingMode = config.get('trading_mode', TradingMode.SPOT)
@@ -152,8 +152,8 @@ class Exchange:
self._ft_has = deep_merge_dicts(self._ft_has, deepcopy(self._ft_has_default)) self._ft_has = deep_merge_dicts(self._ft_has, deepcopy(self._ft_has_default))
if self.trading_mode == TradingMode.FUTURES: if self.trading_mode == TradingMode.FUTURES:
self._ft_has = deep_merge_dicts(self._ft_has_futures, self._ft_has) self._ft_has = deep_merge_dicts(self._ft_has_futures, self._ft_has)
if exchange_config.get('_ft_has_params'): if exchange_conf.get('_ft_has_params'):
self._ft_has = deep_merge_dicts(exchange_config.get('_ft_has_params'), self._ft_has = deep_merge_dicts(exchange_conf.get('_ft_has_params'),
self._ft_has) self._ft_has)
logger.info("Overriding exchange._ft_has with config params, result: %s", self._ft_has) logger.info("Overriding exchange._ft_has with config params, result: %s", self._ft_has)
@@ -165,18 +165,18 @@ class Exchange:
# Initialize ccxt objects # Initialize ccxt objects
ccxt_config = self._ccxt_config ccxt_config = self._ccxt_config
ccxt_config = deep_merge_dicts(exchange_config.get('ccxt_config', {}), ccxt_config) ccxt_config = deep_merge_dicts(exchange_conf.get('ccxt_config', {}), ccxt_config)
ccxt_config = deep_merge_dicts(exchange_config.get('ccxt_sync_config', {}), ccxt_config) ccxt_config = deep_merge_dicts(exchange_conf.get('ccxt_sync_config', {}), ccxt_config)
self._api = self._init_ccxt(exchange_config, ccxt_kwargs=ccxt_config) self._api = self._init_ccxt(exchange_conf, ccxt_kwargs=ccxt_config)
ccxt_async_config = self._ccxt_config ccxt_async_config = self._ccxt_config
ccxt_async_config = deep_merge_dicts(exchange_config.get('ccxt_config', {}), ccxt_async_config = deep_merge_dicts(exchange_conf.get('ccxt_config', {}),
ccxt_async_config) ccxt_async_config)
ccxt_async_config = deep_merge_dicts(exchange_config.get('ccxt_async_config', {}), ccxt_async_config = deep_merge_dicts(exchange_conf.get('ccxt_async_config', {}),
ccxt_async_config) ccxt_async_config)
self._api_async = self._init_ccxt( self._api_async = self._init_ccxt(
exchange_config, ccxt_async, ccxt_kwargs=ccxt_async_config) exchange_conf, ccxt_async, ccxt_kwargs=ccxt_async_config)
logger.info(f'Using Exchange "{self.name}"') logger.info(f'Using Exchange "{self.name}"')
self.required_candle_call_count = 1 self.required_candle_call_count = 1
@@ -189,7 +189,7 @@ class Exchange:
self._startup_candle_count, config.get('timeframe', '')) self._startup_candle_count, config.get('timeframe', ''))
# Converts the interval provided in minutes in config to seconds # Converts the interval provided in minutes in config to seconds
self.markets_refresh_interval: int = exchange_config.get( self.markets_refresh_interval: int = exchange_conf.get(
"markets_refresh_interval", 60) * 60 "markets_refresh_interval", 60) * 60
if self.trading_mode != TradingMode.SPOT and load_leverage_tiers: if self.trading_mode != TradingMode.SPOT and load_leverage_tiers:
freqtrade/freqai/base_models/BasePyTorchRegressor.py
+1
View File
@@ -47,4 +47,5 @@ class BasePyTorchRegressor(BasePyTorchModel):
self.model.model.eval() self.model.model.eval()
y = self.model.model(x) y = self.model.model(x)
pred_df = DataFrame(y.detach().tolist(), columns=[dk.label_list[0]]) pred_df = DataFrame(y.detach().tolist(), columns=[dk.label_list[0]])
pred_df = dk.denormalize_labels_from_metadata(pred_df)
return (pred_df, dk.do_predict) return (pred_df, dk.do_predict)
freqtrade/freqai/prediction_models/PyTorchTransformerRegressor.py
+2 -2
View File
@@ -119,11 +119,11 @@ class PyTorchTransformerRegressor(BasePyTorchRegressor):
x = x.unsqueeze(0) x = x.unsqueeze(0)
# create empty torch tensor # create empty torch tensor
self.model.model.eval() self.model.model.eval()
yb = torch.empty(0) yb = torch.empty(0).to(self.device)
if x.shape[1] > 1: if x.shape[1] > 1:
ws = self.window_size ws = self.window_size
for i in range(0, x.shape[1] - ws): for i in range(0, x.shape[1] - ws):
xb = x[:, i:i + ws, :] xb = x[:, i:i + ws, :].to(self.device)
y = self.model.model(xb) y = self.model.model(xb)
yb = torch.cat((yb, y), dim=0) yb = torch.cat((yb, y), dim=0)
else: else:
freqtrade/freqai/torch/PyTorchTransformerModel.py
+2 -2
View File
@@ -1,7 +1,7 @@
import math import math
import torch import torch
import torch.nn as nn from torch import nn
""" """
@@ -68,7 +68,7 @@ class PyTorchTransformerModel(nn.Module):
return x return x
class PositionalEncoding(torch.nn.Module): class PositionalEncoding(nn.Module):
def __init__(self, d_model, max_len=5000): def __init__(self, d_model, max_len=5000):
""" """
Args Args
freqtrade/freqtradebot.py
+6 -2
View File
@@ -13,7 +13,7 @@ from schedule import Scheduler
from freqtrade import constants from freqtrade import constants
from freqtrade.configuration import validate_config_consistency from freqtrade.configuration import validate_config_consistency
from freqtrade.constants import BuySell, Config, LongShort from freqtrade.constants import BuySell, Config, ExchangeConfig, LongShort
from freqtrade.data.converter import order_book_to_dataframe from freqtrade.data.converter import order_book_to_dataframe
from freqtrade.data.dataprovider import DataProvider from freqtrade.data.dataprovider import DataProvider
from freqtrade.edge import Edge from freqtrade.edge import Edge
@@ -23,6 +23,7 @@ from freqtrade.exceptions import (DependencyException, ExchangeError, Insufficie
InvalidOrderException, PricingError) InvalidOrderException, PricingError)
from freqtrade.exchange import (ROUND_DOWN, ROUND_UP, timeframe_to_minutes, timeframe_to_next_date, from freqtrade.exchange import (ROUND_DOWN, ROUND_UP, timeframe_to_minutes, timeframe_to_next_date,
timeframe_to_seconds) timeframe_to_seconds)
from freqtrade.exchange.common import remove_exchange_credentials
from freqtrade.misc import safe_value_fallback, safe_value_fallback2 from freqtrade.misc import safe_value_fallback, safe_value_fallback2
from freqtrade.mixins import LoggingMixin from freqtrade.mixins import LoggingMixin
from freqtrade.persistence import Order, PairLocks, Trade, init_db from freqtrade.persistence import Order, PairLocks, Trade, init_db
@@ -63,6 +64,9 @@ class FreqtradeBot(LoggingMixin):
# Init objects # Init objects
self.config = config self.config = config
exchange_config: ExchangeConfig = deepcopy(config['exchange'])
# Remove credentials from original exchange config to avoid accidental credentail exposure
remove_exchange_credentials(config['exchange'], True)
self.strategy: IStrategy = StrategyResolver.load_strategy(self.config) self.strategy: IStrategy = StrategyResolver.load_strategy(self.config)
@@ -70,7 +74,7 @@ class FreqtradeBot(LoggingMixin):
validate_config_consistency(config) validate_config_consistency(config)
self.exchange = ExchangeResolver.load_exchange( self.exchange = ExchangeResolver.load_exchange(
self.config, load_leverage_tiers=True) self.config, exchange_config=exchange_config, load_leverage_tiers=True)
init_db(self.config['db_url']) init_db(self.config['db_url'])
freqtrade/resolvers/exchange_resolver.py
+6 -4
View File
@@ -2,9 +2,10 @@
This module loads custom exchanges This module loads custom exchanges
""" """
import logging import logging
from typing import Optional
import freqtrade.exchange as exchanges import freqtrade.exchange as exchanges
from freqtrade.constants import Config from freqtrade.constants import Config, ExchangeConfig
from freqtrade.exchange import MAP_EXCHANGE_CHILDCLASS, Exchange from freqtrade.exchange import MAP_EXCHANGE_CHILDCLASS, Exchange
from freqtrade.resolvers import IResolver from freqtrade.resolvers import IResolver
@@ -19,8 +20,8 @@ class ExchangeResolver(IResolver):
object_type = Exchange object_type = Exchange
@staticmethod @staticmethod
def load_exchange(config: Config, validate: bool = True, def load_exchange(config: Config, *, exchange_config: Optional[ExchangeConfig] = None,
load_leverage_tiers: bool = False) -> Exchange: validate: bool = True, load_leverage_tiers: bool = False) -> Exchange:
""" """
Load the custom class from config parameter Load the custom class from config parameter
:param exchange_name: name of the Exchange to load :param exchange_name: name of the Exchange to load
@@ -37,13 +38,14 @@ class ExchangeResolver(IResolver):
kwargs={ kwargs={
'config': config, 'config': config,
'validate': validate, 'validate': validate,
'exchange_config': exchange_config,
'load_leverage_tiers': load_leverage_tiers} 'load_leverage_tiers': load_leverage_tiers}
) )
except ImportError: except ImportError:
logger.info( logger.info(
f"No {exchange_name} specific subclass found. Using the generic class instead.") f"No {exchange_name} specific subclass found. Using the generic class instead.")
if not exchange: if not exchange:
exchange = Exchange(config, validate=validate) exchange = Exchange(config, validate=validate, exchange_config=exchange_config,)
return exchange return exchange
@staticmethod @staticmethod
freqtrade/rpc/api_server/api_backtest.py
+97 -93
View File
@@ -8,14 +8,16 @@ from fastapi import APIRouter, BackgroundTasks, Depends
from fastapi.exceptions import HTTPException from fastapi.exceptions import HTTPException
from freqtrade.configuration.config_validation import validate_config_consistency from freqtrade.configuration.config_validation import validate_config_consistency
from freqtrade.constants import Config
from freqtrade.data.btanalysis import get_backtest_resultlist, load_and_merge_backtest_result from freqtrade.data.btanalysis import get_backtest_resultlist, load_and_merge_backtest_result
from freqtrade.enums import BacktestState from freqtrade.enums import BacktestState
from freqtrade.exceptions import DependencyException, OperationalException from freqtrade.exceptions import DependencyException, OperationalException
from freqtrade.exchange.common import remove_exchange_credentials
from freqtrade.misc import deep_merge_dicts from freqtrade.misc import deep_merge_dicts
from freqtrade.rpc.api_server.api_schemas import (BacktestHistoryEntry, BacktestRequest, from freqtrade.rpc.api_server.api_schemas import (BacktestHistoryEntry, BacktestRequest,
BacktestResponse) BacktestResponse)
from freqtrade.rpc.api_server.deps import get_config, is_webserver_mode from freqtrade.rpc.api_server.deps import get_config, is_webserver_mode
from freqtrade.rpc.api_server.webserver import ApiServer from freqtrade.rpc.api_server.webserver_bgwork import ApiBG
from freqtrade.rpc.rpc import RPCException from freqtrade.rpc.rpc import RPCException
@@ -25,19 +27,92 @@ logger = logging.getLogger(__name__)
router = APIRouter() router = APIRouter()
def __run_backtest_bg(btconfig: Config):
from freqtrade.optimize.optimize_reports import generate_backtest_stats, store_backtest_stats
from freqtrade.resolvers import StrategyResolver
asyncio.set_event_loop(asyncio.new_event_loop())
try:
# Reload strategy
lastconfig = ApiBG.bt['last_config']
strat = StrategyResolver.load_strategy(btconfig)
validate_config_consistency(btconfig)
if (
not ApiBG.bt['bt']
or lastconfig.get('timeframe') != strat.timeframe
or lastconfig.get('timeframe_detail') != btconfig.get('timeframe_detail')
or lastconfig.get('timerange') != btconfig['timerange']
):
from freqtrade.optimize.backtesting import Backtesting
ApiBG.bt['bt'] = Backtesting(btconfig)
ApiBG.bt['bt'].load_bt_data_detail()
else:
ApiBG.bt['bt'].config = btconfig
ApiBG.bt['bt'].init_backtest()
# Only reload data if timeframe changed.
if (
not ApiBG.bt['data']
or not ApiBG.bt['timerange']
or lastconfig.get('timeframe') != strat.timeframe
or lastconfig.get('timerange') != btconfig['timerange']
):
ApiBG.bt['data'], ApiBG.bt['timerange'] = ApiBG.bt[
'bt'].load_bt_data()
lastconfig['timerange'] = btconfig['timerange']
lastconfig['timeframe'] = strat.timeframe
lastconfig['protections'] = btconfig.get('protections', [])
lastconfig['enable_protections'] = btconfig.get('enable_protections')
lastconfig['dry_run_wallet'] = btconfig.get('dry_run_wallet')
ApiBG.bt['bt'].enable_protections = btconfig.get('enable_protections', False)
ApiBG.bt['bt'].strategylist = [strat]
ApiBG.bt['bt'].results = {}
ApiBG.bt['bt'].load_prior_backtest()
ApiBG.bt['bt'].abort = False
if (ApiBG.bt['bt'].results and
strat.get_strategy_name() in ApiBG.bt['bt'].results['strategy']):
# When previous result hash matches - reuse that result and skip backtesting.
logger.info(f'Reusing result of previous backtest for {strat.get_strategy_name()}')
else:
min_date, max_date = ApiBG.bt['bt'].backtest_one_strategy(
strat, ApiBG.bt['data'], ApiBG.bt['timerange'])
ApiBG.bt['bt'].results = generate_backtest_stats(
ApiBG.bt['data'], ApiBG.bt['bt'].all_results,
min_date=min_date, max_date=max_date)
if btconfig.get('export', 'none') == 'trades':
store_backtest_stats(
btconfig['exportfilename'], ApiBG.bt['bt'].results,
datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
)
logger.info("Backtest finished.")
except (Exception, OperationalException, DependencyException) as e:
logger.exception(f"Backtesting caused an error: {e}")
ApiBG.bt['bt_error'] = str(e)
pass
finally:
ApiBG.bgtask_running = False
@router.post('/backtest', response_model=BacktestResponse, tags=['webserver', 'backtest']) @router.post('/backtest', response_model=BacktestResponse, tags=['webserver', 'backtest'])
async def api_start_backtest( # noqa: C901 async def api_start_backtest(
bt_settings: BacktestRequest, background_tasks: BackgroundTasks, bt_settings: BacktestRequest, background_tasks: BackgroundTasks,
config=Depends(get_config), ws_mode=Depends(is_webserver_mode)): config=Depends(get_config), ws_mode=Depends(is_webserver_mode)):
ApiServer._bt['bt_error'] = None ApiBG.bt['bt_error'] = None
"""Start backtesting if not done so already""" """Start backtesting if not done so already"""
if ApiServer._bgtask_running: if ApiBG.bgtask_running:
raise RPCException('Bot Background task already running') raise RPCException('Bot Background task already running')
if ':' in bt_settings.strategy: if ':' in bt_settings.strategy:
raise HTTPException(status_code=500, detail="base64 encoded strategies are not allowed.") raise HTTPException(status_code=500, detail="base64 encoded strategies are not allowed.")
btconfig = deepcopy(config) btconfig = deepcopy(config)
remove_exchange_credentials(btconfig['exchange'], True)
settings = dict(bt_settings) settings = dict(bt_settings)
if settings.get('freqai', None) is not None: if settings.get('freqai', None) is not None:
settings['freqai'] = dict(settings['freqai']) settings['freqai'] = dict(settings['freqai'])
@@ -54,80 +129,9 @@ async def api_start_backtest( # noqa: C901
# Start backtesting # Start backtesting
# Initialize backtesting object # Initialize backtesting object
def run_backtest():
from freqtrade.optimize.optimize_reports import (generate_backtest_stats,
store_backtest_stats)
from freqtrade.resolvers import StrategyResolver
asyncio.set_event_loop(asyncio.new_event_loop())
try:
# Reload strategy
lastconfig = ApiServer._bt['last_config']
strat = StrategyResolver.load_strategy(btconfig)
validate_config_consistency(btconfig)
if ( background_tasks.add_task(__run_backtest_bg, btconfig=btconfig)
not ApiServer._bt['bt'] ApiBG.bgtask_running = True
or lastconfig.get('timeframe') != strat.timeframe
or lastconfig.get('timeframe_detail') != btconfig.get('timeframe_detail')
or lastconfig.get('timerange') != btconfig['timerange']
):
from freqtrade.optimize.backtesting import Backtesting
ApiServer._bt['bt'] = Backtesting(btconfig)
ApiServer._bt['bt'].load_bt_data_detail()
else:
ApiServer._bt['bt'].config = btconfig
ApiServer._bt['bt'].init_backtest()
# Only reload data if timeframe changed.
if (
not ApiServer._bt['data']
or not ApiServer._bt['timerange']
or lastconfig.get('timeframe') != strat.timeframe
or lastconfig.get('timerange') != btconfig['timerange']
):
ApiServer._bt['data'], ApiServer._bt['timerange'] = ApiServer._bt[
'bt'].load_bt_data()
lastconfig['timerange'] = btconfig['timerange']
lastconfig['timeframe'] = strat.timeframe
lastconfig['protections'] = btconfig.get('protections', [])
lastconfig['enable_protections'] = btconfig.get('enable_protections')
lastconfig['dry_run_wallet'] = btconfig.get('dry_run_wallet')
ApiServer._bt['bt'].enable_protections = btconfig.get('enable_protections', False)
ApiServer._bt['bt'].strategylist = [strat]
ApiServer._bt['bt'].results = {}
ApiServer._bt['bt'].load_prior_backtest()
ApiServer._bt['bt'].abort = False
if (ApiServer._bt['bt'].results and
strat.get_strategy_name() in ApiServer._bt['bt'].results['strategy']):
# When previous result hash matches - reuse that result and skip backtesting.
logger.info(f'Reusing result of previous backtest for {strat.get_strategy_name()}')
else:
min_date, max_date = ApiServer._bt['bt'].backtest_one_strategy(
strat, ApiServer._bt['data'], ApiServer._bt['timerange'])
ApiServer._bt['bt'].results = generate_backtest_stats(
ApiServer._bt['data'], ApiServer._bt['bt'].all_results,
min_date=min_date, max_date=max_date)
if btconfig.get('export', 'none') == 'trades':
store_backtest_stats(
btconfig['exportfilename'], ApiServer._bt['bt'].results,
datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
)
logger.info("Backtest finished.")
except (Exception, OperationalException, DependencyException) as e:
logger.exception(f"Backtesting caused an error: {e}")
ApiServer._bt['bt_error'] = str(e)
pass
finally:
ApiServer._bgtask_running = False
background_tasks.add_task(run_backtest)
ApiServer._bgtask_running = True
return { return {
"status": "running", "status": "running",
@@ -145,18 +149,18 @@ def api_get_backtest(ws_mode=Depends(is_webserver_mode)):
Returns Result after backtesting has been ran. Returns Result after backtesting has been ran.
""" """
from freqtrade.persistence import LocalTrade from freqtrade.persistence import LocalTrade
if ApiServer._bgtask_running: if ApiBG.bgtask_running:
return { return {
"status": "running", "status": "running",
"running": True, "running": True,
"step": (ApiServer._bt['bt'].progress.action if ApiServer._bt['bt'] "step": (ApiBG.bt['bt'].progress.action if ApiBG.bt['bt']
else str(BacktestState.STARTUP)), else str(BacktestState.STARTUP)),
"progress": ApiServer._bt['bt'].progress.progress if ApiServer._bt['bt'] else 0, "progress": ApiBG.bt['bt'].progress.progress if ApiBG.bt['bt'] else 0,
"trade_count": len(LocalTrade.trades), "trade_count": len(LocalTrade.trades),
"status_msg": "Backtest running", "status_msg": "Backtest running",
} }
if not ApiServer._bt['bt']: if not ApiBG.bt['bt']:
return { return {
"status": "not_started", "status": "not_started",
"running": False, "running": False,
@@ -164,13 +168,13 @@ def api_get_backtest(ws_mode=Depends(is_webserver_mode)):
"progress": 0, "progress": 0,
"status_msg": "Backtest not yet executed" "status_msg": "Backtest not yet executed"
} }
if ApiServer._bt['bt_error']: if ApiBG.bt['bt_error']:
return { return {
"status": "error", "status": "error",
"running": False, "running": False,
"step": "", "step": "",
"progress": 0, "progress": 0,
"status_msg": f"Backtest failed with {ApiServer._bt['bt_error']}" "status_msg": f"Backtest failed with {ApiBG.bt['bt_error']}"
} }
return { return {
@@ -179,14 +183,14 @@ def api_get_backtest(ws_mode=Depends(is_webserver_mode)):
"status_msg": "Backtest ended", "status_msg": "Backtest ended",
"step": "finished", "step": "finished",
"progress": 1, "progress": 1,
"backtest_result": ApiServer._bt['bt'].results, "backtest_result": ApiBG.bt['bt'].results,
} }
@router.delete('/backtest', response_model=BacktestResponse, tags=['webserver', 'backtest']) @router.delete('/backtest', response_model=BacktestResponse, tags=['webserver', 'backtest'])
def api_delete_backtest(ws_mode=Depends(is_webserver_mode)): def api_delete_backtest(ws_mode=Depends(is_webserver_mode)):
"""Reset backtesting""" """Reset backtesting"""
if ApiServer._bgtask_running: if ApiBG.bgtask_running:
return { return {
"status": "running", "status": "running",
"running": True, "running": True,
@@ -194,12 +198,12 @@ def api_delete_backtest(ws_mode=Depends(is_webserver_mode)):
"progress": 0, "progress": 0,
"status_msg": "Backtest running", "status_msg": "Backtest running",
} }
if ApiServer._bt['bt']: if ApiBG.bt['bt']:
ApiServer._bt['bt'].cleanup() ApiBG.bt['bt'].cleanup()
del ApiServer._bt['bt'] del ApiBG.bt['bt']
ApiServer._bt['bt'] = None ApiBG.bt['bt'] = None
del ApiServer._bt['data'] del ApiBG.bt['data']
ApiServer._bt['data'] = None ApiBG.bt['data'] = None
logger.info("Backtesting reset") logger.info("Backtesting reset")
return { return {
"status": "reset", "status": "reset",
@@ -212,7 +216,7 @@ def api_delete_backtest(ws_mode=Depends(is_webserver_mode)):
@router.get('/backtest/abort', response_model=BacktestResponse, tags=['webserver', 'backtest']) @router.get('/backtest/abort', response_model=BacktestResponse, tags=['webserver', 'backtest'])
def api_backtest_abort(ws_mode=Depends(is_webserver_mode)): def api_backtest_abort(ws_mode=Depends(is_webserver_mode)):
if not ApiServer._bgtask_running: if not ApiBG.bgtask_running:
return { return {
"status": "not_running", "status": "not_running",
"running": False, "running": False,
@@ -220,7 +224,7 @@ def api_backtest_abort(ws_mode=Depends(is_webserver_mode)):
"progress": 0, "progress": 0,
"status_msg": "Backtest ended", "status_msg": "Backtest ended",
} }
ApiServer._bt['bt'].abort = True ApiBG.bt['bt'].abort = True
return { return {
"status": "stopping", "status": "stopping",
"running": False, "running": False,
freqtrade/rpc/api_server/deps.py
+4 -3
View File
@@ -6,6 +6,7 @@ from fastapi import Depends
from freqtrade.enums import RunMode from freqtrade.enums import RunMode
from freqtrade.persistence import Trade from freqtrade.persistence import Trade
from freqtrade.persistence.models import _request_id_ctx_var from freqtrade.persistence.models import _request_id_ctx_var
from freqtrade.rpc.api_server.webserver_bgwork import ApiBG
from freqtrade.rpc.rpc import RPC, RPCException from freqtrade.rpc.rpc import RPC, RPCException
from .webserver import ApiServer from .webserver import ApiServer
@@ -43,11 +44,11 @@ def get_api_config() -> Dict[str, Any]:
def get_exchange(config=Depends(get_config)): def get_exchange(config=Depends(get_config)):
if not ApiServer._exchange: if not ApiBG.exchange:
from freqtrade.resolvers import ExchangeResolver from freqtrade.resolvers import ExchangeResolver
ApiServer._exchange = ExchangeResolver.load_exchange( ApiBG.exchange = ExchangeResolver.load_exchange(
config, load_leverage_tiers=False) config, load_leverage_tiers=False)
return ApiServer._exchange return ApiBG.exchange
def get_message_stream(): def get_message_stream():
freqtrade/rpc/api_server/webserver.py
+2 -13
View File
@@ -1,6 +1,6 @@
import logging import logging
from ipaddress import IPv4Address from ipaddress import IPv4Address
from typing import Any, Dict, Optional from typing import Any, Optional
import orjson import orjson
import uvicorn import uvicorn
@@ -36,19 +36,8 @@ class ApiServer(RPCHandler):
__initialized = False __initialized = False
_rpc: RPC _rpc: RPC
# Backtesting type: Backtesting
_bt: Dict[str, Any] = {
'bt': None,
'data': None,
'timerange': None,
'last_config': {},
'bt_error': None,
}
_has_rpc: bool = False _has_rpc: bool = False
_bgtask_running: bool = False
_config: Config = {} _config: Config = {}
# Exchange - only available in webserver mode.
_exchange = None
# websocket message stuff # websocket message stuff
_message_stream: Optional[MessageStream] = None _message_stream: Optional[MessageStream] = None
@@ -85,7 +74,7 @@ class ApiServer(RPCHandler):
""" """
Attach rpc handler Attach rpc handler
""" """
if not self._has_rpc: if not ApiServer._has_rpc:
ApiServer._rpc = rpc ApiServer._rpc = rpc
ApiServer._has_rpc = True ApiServer._has_rpc = True
else: else:
freqtrade/rpc/api_server/webserver_bgwork.py
+16
View File
@@ -0,0 +1,16 @@
from typing import Any, Dict
class ApiBG():
# Backtesting type: Backtesting
bt: Dict[str, Any] = {
'bt': None,
'data': None,
'timerange': None,
'last_config': {},
'bt_error': None,
}
bgtask_running: bool = False
# Exchange - only available in webserver mode.
exchange = None
requirements-dev.txt
+1 -1
View File
@@ -9,7 +9,7 @@
coveralls==3.3.1 coveralls==3.3.1
ruff==0.0.267 ruff==0.0.267
mypy==1.3.0 mypy==1.3.0
pre-commit==3.3.1 pre-commit==3.3.2
pytest==7.3.1 pytest==7.3.1
pytest-asyncio==0.21.0 pytest-asyncio==0.21.0
pytest-cov==4.0.0 pytest-cov==4.0.0
requirements.txt
+3 -3
View File
@@ -2,7 +2,7 @@ numpy==1.24.3
pandas==2.0.1 pandas==2.0.1
pandas-ta==0.3.14b pandas-ta==0.3.14b
ccxt==3.0.103 ccxt==3.1.5
cryptography==40.0.2; platform_machine != 'armv7l' cryptography==40.0.2; platform_machine != 'armv7l'
cryptography==40.0.1; platform_machine == 'armv7l' cryptography==40.0.1; platform_machine == 'armv7l'
aiohttp==3.8.4 aiohttp==3.8.4
@@ -11,7 +11,7 @@ python-telegram-bot==20.3
# can't be hard-pinned due to telegram-bot pinning httpx with ~ # can't be hard-pinned due to telegram-bot pinning httpx with ~
httpx>=0.23.3 httpx>=0.23.3
arrow==1.2.3 arrow==1.2.3
cachetools==4.2.2 cachetools==5.3.0
requests==2.30.0 requests==2.30.0
urllib3==2.0.2 urllib3==2.0.2
jsonschema==4.17.3 jsonschema==4.17.3
@@ -38,7 +38,7 @@ orjson==3.8.12
sdnotify==0.3.2 sdnotify==0.3.2
# API Server # API Server
fastapi==0.95.1 fastapi==0.95.2
pydantic==1.10.7 pydantic==1.10.7
uvicorn==0.22.0 uvicorn==0.22.0
pyjwt==2.7.0 pyjwt==2.7.0
tests/exchange/test_exchange.py
+4 -6
View File
@@ -20,7 +20,7 @@ from freqtrade.exchange import (Binance, Bittrex, Exchange, Kraken, amount_to_pr
timeframe_to_minutes, timeframe_to_msecs, timeframe_to_next_date, timeframe_to_minutes, timeframe_to_msecs, timeframe_to_next_date,
timeframe_to_prev_date, timeframe_to_seconds) timeframe_to_prev_date, timeframe_to_seconds)
from freqtrade.exchange.common import (API_FETCH_ORDER_RETRY_COUNT, API_RETRY_COUNT, from freqtrade.exchange.common import (API_FETCH_ORDER_RETRY_COUNT, API_RETRY_COUNT,
calculate_backoff, remove_credentials) calculate_backoff, remove_exchange_credentials)
from freqtrade.exchange.exchange import amount_to_contract_precision from freqtrade.exchange.exchange import amount_to_contract_precision
from freqtrade.resolvers.exchange_resolver import ExchangeResolver from freqtrade.resolvers.exchange_resolver import ExchangeResolver
from tests.conftest import (EXMS, generate_test_data_raw, get_mock_coro, get_patched_exchange, from tests.conftest import (EXMS, generate_test_data_raw, get_mock_coro, get_patched_exchange,
@@ -137,16 +137,14 @@ def test_init(default_conf, mocker, caplog):
assert log_has('Instance is running with dry_run enabled', caplog) assert log_has('Instance is running with dry_run enabled', caplog)
def test_remove_credentials(default_conf, caplog) -> None: def test_remove_exchange_credentials(default_conf) -> None:
conf = deepcopy(default_conf) conf = deepcopy(default_conf)
conf['dry_run'] = False remove_exchange_credentials(conf['exchange'], False)
remove_credentials(conf)
assert conf['exchange']['key'] != '' assert conf['exchange']['key'] != ''
assert conf['exchange']['secret'] != '' assert conf['exchange']['secret'] != ''
conf['dry_run'] = True remove_exchange_credentials(conf['exchange'], True)
remove_credentials(conf)
assert conf['exchange']['key'] == '' assert conf['exchange']['key'] == ''
assert conf['exchange']['secret'] == '' assert conf['exchange']['secret'] == ''
assert conf['exchange']['password'] == '' assert conf['exchange']['password'] == ''
tests/rpc/test_rpc_apiserver.py
+115 -110
View File
@@ -21,11 +21,13 @@ from freqtrade.__init__ import __version__
from freqtrade.enums import CandleType, RunMode, State, TradingMode from freqtrade.enums import CandleType, RunMode, State, TradingMode
from freqtrade.exceptions import DependencyException, ExchangeError, OperationalException from freqtrade.exceptions import DependencyException, ExchangeError, OperationalException
from freqtrade.loggers import setup_logging, setup_logging_pre from freqtrade.loggers import setup_logging, setup_logging_pre
from freqtrade.optimize.backtesting import Backtesting
from freqtrade.persistence import PairLocks, Trade from freqtrade.persistence import PairLocks, Trade
from freqtrade.rpc import RPC from freqtrade.rpc import RPC
from freqtrade.rpc.api_server import ApiServer from freqtrade.rpc.api_server import ApiServer
from freqtrade.rpc.api_server.api_auth import create_token, get_user_from_token from freqtrade.rpc.api_server.api_auth import create_token, get_user_from_token
from freqtrade.rpc.api_server.uvicorn_threaded import UvicornServer from freqtrade.rpc.api_server.uvicorn_threaded import UvicornServer
from freqtrade.rpc.api_server.webserver_bgwork import ApiBG
from tests.conftest import (CURRENT_TEST_STRATEGY, EXMS, create_mock_trades, get_mock_coro, from tests.conftest import (CURRENT_TEST_STRATEGY, EXMS, create_mock_trades, get_mock_coro,
get_patched_freqtradebot, log_has, log_has_re, patch_get_signal) get_patched_freqtradebot, log_has, log_has_re, patch_get_signal)
@@ -1665,137 +1667,140 @@ def test_sysinfo(botclient):
def test_api_backtesting(botclient, mocker, fee, caplog, tmpdir): def test_api_backtesting(botclient, mocker, fee, caplog, tmpdir):
ftbot, client = botclient try:
mocker.patch(f'{EXMS}.get_fee', fee) ftbot, client = botclient
mocker.patch(f'{EXMS}.get_fee', fee)
rc = client_get(client, f"{BASE_URI}/backtest") rc = client_get(client, f"{BASE_URI}/backtest")
# Backtest prevented in default mode # Backtest prevented in default mode
assert_response(rc, 502) assert_response(rc, 502)
ftbot.config['runmode'] = RunMode.WEBSERVER ftbot.config['runmode'] = RunMode.WEBSERVER
# Backtesting not started yet # Backtesting not started yet
rc = client_get(client, f"{BASE_URI}/backtest") rc = client_get(client, f"{BASE_URI}/backtest")
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'not_started' assert result['status'] == 'not_started'
assert not result['running'] assert not result['running']
assert result['status_msg'] == 'Backtest not yet executed' assert result['status_msg'] == 'Backtest not yet executed'
assert result['progress'] == 0 assert result['progress'] == 0
# Reset backtesting # Reset backtesting
rc = client_delete(client, f"{BASE_URI}/backtest") rc = client_delete(client, f"{BASE_URI}/backtest")
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'reset' assert result['status'] == 'reset'
assert not result['running'] assert not result['running']
assert result['status_msg'] == 'Backtest reset' assert result['status_msg'] == 'Backtest reset'
ftbot.config['export'] = 'trades' ftbot.config['export'] = 'trades'
ftbot.config['backtest_cache'] = 'day' ftbot.config['backtest_cache'] = 'day'
ftbot.config['user_data_dir'] = Path(tmpdir) ftbot.config['user_data_dir'] = Path(tmpdir)
ftbot.config['exportfilename'] = Path(tmpdir) / "backtest_results" ftbot.config['exportfilename'] = Path(tmpdir) / "backtest_results"
ftbot.config['exportfilename'].mkdir() ftbot.config['exportfilename'].mkdir()
# start backtesting # start backtesting
data = { data = {
"strategy": CURRENT_TEST_STRATEGY, "strategy": CURRENT_TEST_STRATEGY,
"timeframe": "5m", "timeframe": "5m",
"timerange": "20180110-20180111", "timerange": "20180110-20180111",
"max_open_trades": 3, "max_open_trades": 3,
"stake_amount": 100, "stake_amount": 100,
"dry_run_wallet": 1000, "dry_run_wallet": 1000,
"enable_protections": False "enable_protections": False
} }
rc = client_post(client, f"{BASE_URI}/backtest", data=data) rc = client_post(client, f"{BASE_URI}/backtest", data=data)
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'running' assert result['status'] == 'running'
assert result['progress'] == 0 assert result['progress'] == 0
assert result['running'] assert result['running']
assert result['status_msg'] == 'Backtest started' assert result['status_msg'] == 'Backtest started'
rc = client_get(client, f"{BASE_URI}/backtest") rc = client_get(client, f"{BASE_URI}/backtest")
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'ended' assert result['status'] == 'ended'
assert not result['running'] assert not result['running']
assert result['status_msg'] == 'Backtest ended' assert result['status_msg'] == 'Backtest ended'
assert result['progress'] == 1 assert result['progress'] == 1
assert result['backtest_result'] assert result['backtest_result']
rc = client_get(client, f"{BASE_URI}/backtest/abort") rc = client_get(client, f"{BASE_URI}/backtest/abort")
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'not_running' assert result['status'] == 'not_running'
assert not result['running'] assert not result['running']
assert result['status_msg'] == 'Backtest ended' assert result['status_msg'] == 'Backtest ended'
# Simulate running backtest # Simulate running backtest
ApiServer._bgtask_running = True ApiBG.bgtask_running = True
rc = client_get(client, f"{BASE_URI}/backtest/abort") rc = client_get(client, f"{BASE_URI}/backtest/abort")
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'stopping' assert result['status'] == 'stopping'
assert not result['running'] assert not result['running']
assert result['status_msg'] == 'Backtest ended' assert result['status_msg'] == 'Backtest ended'
# Get running backtest... # Get running backtest...
rc = client_get(client, f"{BASE_URI}/backtest") rc = client_get(client, f"{BASE_URI}/backtest")
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'running' assert result['status'] == 'running'
assert result['running'] assert result['running']
assert result['step'] == "backtest" assert result['step'] == "backtest"
assert result['status_msg'] == "Backtest running" assert result['status_msg'] == "Backtest running"
# Try delete with task still running # Try delete with task still running
rc = client_delete(client, f"{BASE_URI}/backtest") rc = client_delete(client, f"{BASE_URI}/backtest")
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'running' assert result['status'] == 'running'
# Post to backtest that's still running # Post to backtest that's still running
rc = client_post(client, f"{BASE_URI}/backtest", data=data) rc = client_post(client, f"{BASE_URI}/backtest", data=data)
assert_response(rc, 502) assert_response(rc, 502)
result = rc.json() result = rc.json()
assert 'Bot Background task already running' in result['error'] assert 'Bot Background task already running' in result['error']
ApiServer._bgtask_running = False ApiBG.bgtask_running = False
# Rerun backtest (should get previous result) # Rerun backtest (should get previous result)
rc = client_post(client, f"{BASE_URI}/backtest", data=data) rc = client_post(client, f"{BASE_URI}/backtest", data=data)
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert log_has_re('Reusing result of previous backtest.*', caplog) assert log_has_re('Reusing result of previous backtest.*', caplog)
data['stake_amount'] = 101 data['stake_amount'] = 101
mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest_one_strategy', mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest_one_strategy',
side_effect=DependencyException('DeadBeef')) side_effect=DependencyException('DeadBeef'))
rc = client_post(client, f"{BASE_URI}/backtest", data=data) rc = client_post(client, f"{BASE_URI}/backtest", data=data)
assert log_has("Backtesting caused an error: DeadBeef", caplog) assert log_has("Backtesting caused an error: DeadBeef", caplog)
rc = client_get(client, f"{BASE_URI}/backtest") rc = client_get(client, f"{BASE_URI}/backtest")
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'error' assert result['status'] == 'error'
assert 'Backtest failed' in result['status_msg'] assert 'Backtest failed' in result['status_msg']
# Delete backtesting to avoid leakage since the backtest-object may stick around. # Delete backtesting to avoid leakage since the backtest-object may stick around.
rc = client_delete(client, f"{BASE_URI}/backtest") rc = client_delete(client, f"{BASE_URI}/backtest")
assert_response(rc) assert_response(rc)
result = rc.json() result = rc.json()
assert result['status'] == 'reset' assert result['status'] == 'reset'
assert not result['running'] assert not result['running']
assert result['status_msg'] == 'Backtest reset' assert result['status_msg'] == 'Backtest reset'
# Disallow base64 strategies # Disallow base64 strategies
data['strategy'] = "xx:cHJpbnQoImhlbGxvIHdvcmxkIik=" data['strategy'] = "xx:cHJpbnQoImhlbGxvIHdvcmxkIik="
rc = client_post(client, f"{BASE_URI}/backtest", data=data) rc = client_post(client, f"{BASE_URI}/backtest", data=data)
assert_response(rc, 500) assert_response(rc, 500)
finally:
Backtesting.cleanup()
def test_api_backtest_history(botclient, mocker, testdatadir): def test_api_backtest_history(botclient, mocker, testdatadir):
tests/test_freqtradebot.py
+30 -2
View File
@@ -121,7 +121,7 @@ def test_order_dict(default_conf_usdt, mocker, runmode, caplog) -> None:
freqtrade = FreqtradeBot(conf) freqtrade = FreqtradeBot(conf)
if runmode == RunMode.LIVE: if runmode == RunMode.LIVE:
assert not log_has_re(".*stoploss_on_exchange .* dry-run", caplog) assert not log_has_re(r".*stoploss_on_exchange .* dry-run", caplog)
assert freqtrade.strategy.order_types['stoploss_on_exchange'] assert freqtrade.strategy.order_types['stoploss_on_exchange']
caplog.clear() caplog.clear()
@@ -136,7 +136,7 @@ def test_order_dict(default_conf_usdt, mocker, runmode, caplog) -> None:
} }
freqtrade = FreqtradeBot(conf) freqtrade = FreqtradeBot(conf)
assert not freqtrade.strategy.order_types['stoploss_on_exchange'] assert not freqtrade.strategy.order_types['stoploss_on_exchange']
assert not log_has_re(".*stoploss_on_exchange .* dry-run", caplog) assert not log_has_re(r".*stoploss_on_exchange .* dry-run", caplog)
def test_get_trade_stake_amount(default_conf_usdt, mocker) -> None: def test_get_trade_stake_amount(default_conf_usdt, mocker) -> None:
@@ -149,6 +149,34 @@ def test_get_trade_stake_amount(default_conf_usdt, mocker) -> None:
assert result == default_conf_usdt['stake_amount'] assert result == default_conf_usdt['stake_amount']
@pytest.mark.parametrize('runmode', [
RunMode.DRY_RUN,
RunMode.LIVE
])
def test_load_strategy_no_keys(default_conf_usdt, mocker, runmode, caplog) -> None:
patch_RPCManager(mocker)
patch_exchange(mocker)
conf = deepcopy(default_conf_usdt)
conf['runmode'] = runmode
erm = mocker.patch('freqtrade.freqtradebot.ExchangeResolver.load_exchange')
freqtrade = FreqtradeBot(conf)
strategy_config = freqtrade.strategy.config
assert id(strategy_config['exchange']) == id(conf['exchange'])
# Keys have been removed and are not passed to the exchange
assert strategy_config['exchange']['key'] == ''
assert strategy_config['exchange']['secret'] == ''
assert erm.call_count == 1
ex_conf = erm.call_args_list[0][1]['exchange_config']
assert id(ex_conf) != id(conf['exchange'])
# Keys are still present
assert ex_conf['key'] != ''
assert ex_conf['key'] == default_conf_usdt['exchange']['key']
assert ex_conf['secret'] != ''
assert ex_conf['secret'] == default_conf_usdt['exchange']['secret']
@pytest.mark.parametrize("amend_last,wallet,max_open,lsamr,expected", [ @pytest.mark.parametrize("amend_last,wallet,max_open,lsamr,expected", [
(False, 120, 2, 0.5, [60, None]), (False, 120, 2, 0.5, [60, None]),
(True, 120, 2, 0.5, [60, 58.8]), (True, 120, 2, 0.5, [60, 58.8]),