diff --git a/docs/backtesting.md b/docs/backtesting.md index 90256283e..a49db47c4 100644 --- a/docs/backtesting.md +++ b/docs/backtesting.md @@ -248,7 +248,7 @@ All listed Strategies need to be in the same directory. freqtrade backtesting --timerange 20180401-20180410 --ticker-interval 5m --strategy-list Strategy001 Strategy002 --export trades ``` -This will save the results to `user_data/backtest_data/backtest-result-.json`, injecting the strategy-name into the target filename. +This will save the results to `user_data/backtest_results/backtest-result-.json`, injecting the strategy-name into the target filename. There will be an additional table comparing win/losses of the different strategies (identical to the "Total" row in the first table). Detailed output for all strategies one after the other will be available, so make sure to scroll up. diff --git a/docs/bot-usage.md b/docs/bot-usage.md index 2873f5e8f..27451bcdb 100644 --- a/docs/bot-usage.md +++ b/docs/bot-usage.md @@ -9,38 +9,43 @@ This page explains the different parameters of the bot and how to run it. ## Bot commands ``` -usage: freqtrade [-h] [-v] [--logfile FILE] [--version] [-c PATH] [-d PATH] - [-s NAME] [--strategy-path PATH] [--db-url PATH] - [--sd-notify] - {backtesting,edge,hyperopt} ... +usage: freqtrade [-h] [-v] [--logfile FILE] [-V] [-c PATH] [-d PATH] + [--userdir PATH] [-s NAME] [--strategy-path PATH] + [--db-url PATH] [--sd-notify] + {backtesting,edge,hyperopt,create-userdir,list-exchanges} ... Free, open source crypto trading bot positional arguments: - {backtesting,edge,hyperopt} + {backtesting,edge,hyperopt,create-userdir,list-exchanges} backtesting Backtesting module. edge Edge module. hyperopt Hyperopt module. + create-userdir Create user-data directory. + list-exchanges Print available exchanges. optional arguments: -h, --help show this help message and exit -v, --verbose Verbose mode (-vv for more, -vvv to get all messages). - --logfile FILE Log to the file specified + --logfile FILE Log to the file specified. -V, --version show program's version number and exit -c PATH, --config PATH - Specify configuration file (default: None). Multiple - --config options may be used. Can be set to '-' to - read config from stdin. + Specify configuration file (default: `config.json`). + Multiple --config options may be used. Can be set to + `-` to read config from stdin. -d PATH, --datadir PATH - Path to backtest data. + Path to directory with historical backtesting data. + --userdir PATH, --user-data-dir PATH + Path to userdata directory. -s NAME, --strategy NAME Specify strategy class name (default: - DefaultStrategy). + `DefaultStrategy`). --strategy-path PATH Specify additional strategy lookup path. - --db-url PATH Override trades database URL, this is useful if - dry_run is enabled or in custom deployments (default: - None). + --db-url PATH Override trades database URL, this is useful in custom + deployments (default: `sqlite:///tradesv3.sqlite` for + Live Run mode, `sqlite://` for Dry Run). --sd-notify Notify systemd service manager. + ``` ### How to specify which configuration file be used? @@ -85,6 +90,29 @@ of your configuration in the project issues or in the Internet. See more details on this technique with examples in the documentation page on [configuration](configuration.md). +### Where to store custom data + +Freqtrade allows the creation of a user-data directory using `freqtrade create-userdir --userdir someDirectory`. +This directory will look as follows: + +``` +user_data/ +├── backtest_results +├── data +├── hyperopts +├── hyperopts_results +├── plot +└── strategies +``` + +You can add the entry "user_data_dir" setting to your configuration, to always point your bot to this directory. +Alternatively, pass in `--userdir` to every command. +The bot will fail to start if the directory does not exist, but will create necessary subdirectories. + +This directory should contain your custom strategies, custom hyperopts and hyperopt loss functions, backtesting historical data (downloaded using either backtesting command or the download script) and plot outputs. + +It is recommended to use version control to keep track of changes to your strategies. + ### How to use **--strategy**? This parameter will allow you to load your custom strategy class. @@ -179,8 +207,8 @@ optional arguments: --export-filename PATH Save backtest results to this filename requires --export to be set as well Example --export- - filename=user_data/backtest_data/backtest_today.json - (default: user_data/backtest_data/backtest- + filename=user_data/backtest_results/backtest_today.json + (default: user_data/backtest_results/backtest- result.json) ``` diff --git a/docs/configuration.md b/docs/configuration.md index c277e1c06..2d21de942 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -91,6 +91,7 @@ Mandatory parameters are marked as **Required**. | `internals.process_throttle_secs` | 5 | **Required.** Set the process throttle. Value in second. | `internals.sd_notify` | false | Enables use of the sd_notify protocol to tell systemd service manager about changes in the bot state and issue keep-alive pings. See [here](installation.md#7-optional-configure-freqtrade-as-a-systemd-service) for more details. | `logfile` | | Specify Logfile. Uses a rolling strategy of 10 files, with 1Mb per file. +| `user_data_dir` | cwd()/user_data | Directory containing user data. Defaults to `./user_data/`. ### Parameters in the strategy diff --git a/docs/data-analysis.md b/docs/data-analysis.md index c89353cc8..6d7d0eb00 100644 --- a/docs/data-analysis.md +++ b/docs/data-analysis.md @@ -11,7 +11,7 @@ You can analyze the results of backtests and trading history easily using Jupyte ```python from freqtrade.data.btanalysis import load_backtest_data # Load backtest results -df = load_backtest_data("user_data/backtest_data/backtest-result.json") +df = load_backtest_data("user_data/backtest_results/backtest-result.json") # Show value-counts per pair df.groupby("pair")["sell_reason"].value_counts() diff --git a/docs/plotting.md b/docs/plotting.md index b8e041d61..acab60b23 100644 --- a/docs/plotting.md +++ b/docs/plotting.md @@ -64,7 +64,7 @@ python3 scripts/plot_dataframe.py --db-url sqlite:///tradesv3.dry_run.sqlite -p To plot trades from a backtesting result, use `--export-filename ` ``` bash -python3 scripts/plot_dataframe.py --export-filename user_data/backtest_data/backtest-result.json -p BTC/ETH +python3 scripts/plot_dataframe.py --export-filename user_data/backtest_results/backtest-result.json -p BTC/ETH ``` To plot a custom strategy the strategy should have first be backtested. diff --git a/freqtrade/configuration/arguments.py b/freqtrade/configuration/arguments.py index c45e3d7ba..ac5ea9716 100644 --- a/freqtrade/configuration/arguments.py +++ b/freqtrade/configuration/arguments.py @@ -7,7 +7,7 @@ from typing import List, Optional from freqtrade.configuration.cli_options import AVAILABLE_CLI_OPTIONS from freqtrade import constants -ARGS_COMMON = ["verbosity", "logfile", "version", "config", "datadir"] +ARGS_COMMON = ["verbosity", "logfile", "version", "config", "datadir", "user_data_dir"] ARGS_STRATEGY = ["strategy", "strategy_path"] @@ -30,6 +30,8 @@ ARGS_EDGE = ARGS_COMMON_OPTIMIZE + ["stoploss_range"] ARGS_LIST_EXCHANGES = ["print_one_column"] +ARGS_CREATE_USERDIR = ["user_data_dir"] + ARGS_DOWNLOAD_DATA = ["pairs", "pairs_file", "days", "exchange", "timeframes", "erase"] ARGS_PLOT_DATAFRAME = (ARGS_COMMON + ARGS_STRATEGY + @@ -98,7 +100,7 @@ class Arguments(object): :return: None """ from freqtrade.optimize import start_backtesting, start_hyperopt, start_edge - from freqtrade.utils import start_download_data, start_list_exchanges + from freqtrade.utils import start_create_userdir, start_download_data, start_list_exchanges subparsers = self.parser.add_subparsers(dest='subparser') @@ -117,6 +119,11 @@ class Arguments(object): hyperopt_cmd.set_defaults(func=start_hyperopt) self._build_args(optionlist=ARGS_HYPEROPT, parser=hyperopt_cmd) + create_userdir_cmd = subparsers.add_parser('create-userdir', + help="Create user-data directory.") + create_userdir_cmd.set_defaults(func=start_create_userdir) + self._build_args(optionlist=ARGS_CREATE_USERDIR, parser=create_userdir_cmd) + # Add list-exchanges subcommand list_exchanges_cmd = subparsers.add_parser( 'list-exchanges', diff --git a/freqtrade/configuration/cli_options.py b/freqtrade/configuration/cli_options.py index d39013737..48a0147fd 100644 --- a/freqtrade/configuration/cli_options.py +++ b/freqtrade/configuration/cli_options.py @@ -55,7 +55,12 @@ AVAILABLE_CLI_OPTIONS = { ), "datadir": Arg( '-d', '--datadir', - help='Path to backtest data.', + help='Path to directory with historical backtesting data.', + metavar='PATH', + ), + "user_data_dir": Arg( + '--userdir', '--user-data-dir', + help='Path to userdata directory.', metavar='PATH', ), # Main options @@ -146,9 +151,9 @@ AVAILABLE_CLI_OPTIONS = { '--export-filename', help='Save backtest results to the file with this filename (default: `%(default)s`). ' 'Requires `--export` to be set as well. ' - 'Example: `--export-filename=user_data/backtest_data/backtest_today.json`', + 'Example: `--export-filename=user_data/backtest_results/backtest_today.json`', metavar='PATH', - default=os.path.join('user_data', 'backtest_data', + default=os.path.join('user_data', 'backtest_results', 'backtest-result.json'), ), # Edge diff --git a/freqtrade/configuration/configuration.py b/freqtrade/configuration/configuration.py index b3f54d59e..47f8cf43b 100644 --- a/freqtrade/configuration/configuration.py +++ b/freqtrade/configuration/configuration.py @@ -7,11 +7,12 @@ from argparse import Namespace from pathlib import Path from typing import Any, Callable, Dict, List, Optional -from freqtrade import constants, OperationalException +from freqtrade import OperationalException, constants from freqtrade.configuration.check_exchange import check_exchange -from freqtrade.configuration.create_datadir import create_datadir -from freqtrade.configuration.config_validation import (validate_config_schema, - validate_config_consistency) +from freqtrade.configuration.config_validation import ( + validate_config_consistency, validate_config_schema) +from freqtrade.configuration.directory_operations import (create_datadir, + create_userdata_dir) from freqtrade.configuration.load_config import load_config_file from freqtrade.loggers import setup_logging from freqtrade.misc import deep_merge_dicts, json_load @@ -115,7 +116,9 @@ class Configuration(object): setup_logging(config) - def _process_strategy_options(self, config: Dict[str, Any]) -> None: + def _process_common_options(self, config: Dict[str, Any]) -> None: + + self._process_logging_options(config) # Set strategy if not specified in config and or if it's non default if self.args.strategy != constants.DEFAULT_STRATEGY or not config.get('strategy'): @@ -124,11 +127,6 @@ class Configuration(object): self._args_to_config(config, argname='strategy_path', logstring='Using additional Strategy lookup path: {}') - def _process_common_options(self, config: Dict[str, Any]) -> None: - - self._process_logging_options(config) - self._process_strategy_options(config) - if ('db_url' in self.args and self.args.db_url and self.args.db_url != constants.DEFAULT_DB_PROD_URL): config.update({'db_url': self.args.db_url}) @@ -159,9 +157,19 @@ class Configuration(object): def _process_datadir_options(self, config: Dict[str, Any]) -> None: """ - Extract information for sys.argv and load datadir configuration: - the --datadir option + Extract information for sys.argv and load directory configurations + --user-data, --datadir """ + if 'user_data_dir' in self.args and self.args.user_data_dir: + config.update({'user_data_dir': self.args.user_data_dir}) + elif 'user_data_dir' not in config: + # Default to cwd/user_data (legacy option ...) + config.update({'user_data_dir': str(Path.cwd() / "user_data")}) + + # reset to user_data_dir so this contains the absolute path. + config['user_data_dir'] = create_userdata_dir(config['user_data_dir'], create_dir=False) + logger.info('Using user-data directory: %s ...', config['user_data_dir']) + if 'datadir' in self.args and self.args.datadir: config.update({'datadir': create_datadir(config, self.args.datadir)}) else: @@ -357,7 +365,6 @@ class Configuration(object): else: # Fall back to /dl_path/pairs.json pairs_file = Path(config['datadir']) / config['exchange']['name'].lower() / "pairs.json" - print(config['datadir']) if pairs_file.exists(): with pairs_file.open('r') as f: config['pairs'] = json_load(f) diff --git a/freqtrade/configuration/create_datadir.py b/freqtrade/configuration/create_datadir.py deleted file mode 100644 index acc3a29ca..000000000 --- a/freqtrade/configuration/create_datadir.py +++ /dev/null @@ -1,20 +0,0 @@ -import logging -from typing import Any, Dict, Optional -from pathlib import Path - - -logger = logging.getLogger(__name__) - - -def create_datadir(config: Dict[str, Any], datadir: Optional[str] = None) -> str: - - folder = Path(datadir) if datadir else Path('user_data/data') - if not datadir: - # set datadir - exchange_name = config.get('exchange', {}).get('name').lower() - folder = folder.joinpath(exchange_name) - - if not folder.is_dir(): - folder.mkdir(parents=True) - logger.info(f'Created data directory: {datadir}') - return str(folder) diff --git a/freqtrade/configuration/directory_operations.py b/freqtrade/configuration/directory_operations.py new file mode 100644 index 000000000..395accd90 --- /dev/null +++ b/freqtrade/configuration/directory_operations.py @@ -0,0 +1,50 @@ +import logging +from typing import Any, Dict, Optional +from pathlib import Path + +from freqtrade import OperationalException + +logger = logging.getLogger(__name__) + + +def create_datadir(config: Dict[str, Any], datadir: Optional[str] = None) -> str: + + folder = Path(datadir) if datadir else Path(f"{config['user_data_dir']}/data") + if not datadir: + # set datadir + exchange_name = config.get('exchange', {}).get('name').lower() + folder = folder.joinpath(exchange_name) + + if not folder.is_dir(): + folder.mkdir(parents=True) + logger.info(f'Created data directory: {datadir}') + return str(folder) + + +def create_userdata_dir(directory: str, create_dir=False) -> Path: + """ + Create userdata directory structure. + if create_dir is True, then the parent-directory will be created if it does not exist. + Sub-directories will always be created if the parent directory exists. + Raises OperationalException if given a non-existing directory. + :param directory: Directory to check + :param create_dir: Create directory if it does not exist. + :return: Path object containing the directory + """ + sub_dirs = ["backtest_results", "data", "hyperopts", "hyperopt_results", "plot", "strategies", ] + folder = Path(directory) + if not folder.is_dir(): + if create_dir: + folder.mkdir(parents=True) + logger.info(f'Created user-data directory: {folder}') + else: + raise OperationalException( + f"Directory `{folder}` does not exist. " + "Please use `freqtrade create-userdir` to create a user directory") + + # Create required subdirectories + for f in sub_dirs: + subfolder = folder / f + if not subfolder.is_dir(): + subfolder.mkdir(parents=False) + return folder diff --git a/freqtrade/optimize/__init__.py b/freqtrade/optimize/__init__.py index 8b548eefe..2c7c42c4d 100644 --- a/freqtrade/optimize/__init__.py +++ b/freqtrade/optimize/__init__.py @@ -64,14 +64,14 @@ def start_hyperopt(args: Namespace) -> None: :return: None """ # Import here to avoid loading hyperopt module when it's not used - from freqtrade.optimize.hyperopt import Hyperopt, HYPEROPT_LOCKFILE + from freqtrade.optimize.hyperopt import Hyperopt # Initialize configuration config = setup_configuration(args, RunMode.HYPEROPT) logger.info('Starting freqtrade in Hyperopt mode') - lock = FileLock(HYPEROPT_LOCKFILE) + lock = FileLock(Hyperopt.get_lock_filename(config)) try: with lock.acquire(timeout=1): diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 05b12e653..cf4740f1c 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -5,7 +5,6 @@ This module contains the hyperopt logic """ import logging -import os import sys from collections import OrderedDict @@ -36,9 +35,6 @@ logger = logging.getLogger(__name__) INITIAL_POINTS = 30 MAX_LOSS = 100000 # just a big enough number to be bad result in loss optimization -TICKERDATA_PICKLE = os.path.join('user_data', 'hyperopt_tickerdata.pkl') -TRIALSDATA_PICKLE = os.path.join('user_data', 'hyperopt_results.pickle') -HYPEROPT_LOCKFILE = os.path.join('user_data', 'hyperopt.lock') class Hyperopt(Backtesting): @@ -56,7 +52,12 @@ class Hyperopt(Backtesting): self.custom_hyperoptloss = HyperOptLossResolver(self.config).hyperoptloss self.calculate_loss = self.custom_hyperoptloss.hyperopt_loss_function + self.trials_file = (self.config['user_data_dir'] / + 'hyperopt_results' / 'hyperopt_results.pickle') + self.tickerdata_pickle = (self.config['user_data_dir'] / + 'hyperopt_results' / 'hyperopt_tickerdata.pkl') self.total_epochs = config.get('epochs', 0) + self.current_best_loss = 100 if not self.config.get('hyperopt_continue'): @@ -65,7 +66,6 @@ class Hyperopt(Backtesting): logger.info("Continuing on previous hyperopt results.") # Previous evaluations - self.trials_file = TRIALSDATA_PICKLE self.trials: List = [] # Populate functions here (hasattr is slow so should not be run during "regular" operations) @@ -89,11 +89,16 @@ class Hyperopt(Backtesting): self.config['experimental'] = {} self.config['experimental']['use_sell_signal'] = True + @staticmethod + def get_lock_filename(config) -> str: + + return str(config['user_data_dir'] / 'hyperopt.lock') + def clean_hyperopt(self): """ Remove hyperopt pickle files to restart hyperopt. """ - for f in [TICKERDATA_PICKLE, TRIALSDATA_PICKLE]: + for f in [self.tickerdata_pickle, self.trials_file]: p = Path(f) if p.is_file(): logger.info(f"Removing `{p}`.") @@ -126,7 +131,7 @@ class Hyperopt(Backtesting): """ logger.info('Reading Trials from \'%s\'', self.trials_file) trials = load(self.trials_file) - os.remove(self.trials_file) + self.trials_file.unlink() return trials def log_trials_result(self) -> None: @@ -255,7 +260,7 @@ class Hyperopt(Backtesting): if self.has_space('stoploss'): self.strategy.stoploss = params['stoploss'] - processed = load(TICKERDATA_PICKLE) + processed = load(self.tickerdata_pickle) min_date, max_date = get_timeframe(processed) @@ -327,7 +332,7 @@ class Hyperopt(Backtesting): def load_previous_results(self): """ read trials file if we have one """ - if os.path.exists(self.trials_file) and os.path.getsize(self.trials_file) > 0: + if self.trials_file.is_file() and self.trials_file.stat().st_size > 0: self.trials = self.read_trials() logger.info( 'Loaded %d previous evaluations from disk.', @@ -364,7 +369,7 @@ class Hyperopt(Backtesting): preprocessed = self.strategy.tickerdata_to_dataframe(data) - dump(preprocessed, TICKERDATA_PICKLE) + dump(preprocessed, self.tickerdata_pickle) # We don't need exchange instance anymore while running hyperopt self.exchange = None # type: ignore diff --git a/freqtrade/plot/plotting.py b/freqtrade/plot/plotting.py index e6da581a4..52314be39 100644 --- a/freqtrade/plot/plotting.py +++ b/freqtrade/plot/plotting.py @@ -308,7 +308,7 @@ def generate_plot_filename(pair, ticker_interval) -> str: return file_name -def store_plot_file(fig, filename: str, auto_open: bool = False) -> None: +def store_plot_file(fig, filename: str, directory: Path, auto_open: bool = False) -> None: """ Generate a plot html file from pre populated fig plotly object :param fig: Plotly Figure to plot @@ -316,9 +316,9 @@ def store_plot_file(fig, filename: str, auto_open: bool = False) -> None: :param ticker_interval: Used as part of the filename :return: None """ + directory.mkdir(parents=True, exist_ok=True) - Path("user_data/plots").mkdir(parents=True, exist_ok=True) - _filename = Path('user_data/plots').joinpath(filename) + _filename = directory.joinpath(filename) plot(fig, filename=str(_filename), auto_open=auto_open) logger.info(f"Stored plot as {_filename}") diff --git a/freqtrade/resolvers/hyperopt_resolver.py b/freqtrade/resolvers/hyperopt_resolver.py index 5027d7ddf..15d1997ef 100644 --- a/freqtrade/resolvers/hyperopt_resolver.py +++ b/freqtrade/resolvers/hyperopt_resolver.py @@ -31,7 +31,8 @@ class HyperOptResolver(IResolver): # Verify the hyperopt is in the configuration, otherwise fallback to the default hyperopt hyperopt_name = config.get('hyperopt') or DEFAULT_HYPEROPT - self.hyperopt = self._load_hyperopt(hyperopt_name, extra_dir=config.get('hyperopt_path')) + self.hyperopt = self._load_hyperopt(hyperopt_name, config, + extra_dir=config.get('hyperopt_path')) # Assign ticker_interval to be used in hyperopt self.hyperopt.__class__.ticker_interval = str(config['ticker_interval']) @@ -44,17 +45,18 @@ class HyperOptResolver(IResolver): "Using populate_sell_trend from DefaultStrategy.") def _load_hyperopt( - self, hyperopt_name: str, extra_dir: Optional[str] = None) -> IHyperOpt: + self, hyperopt_name: str, config: Dict, extra_dir: Optional[str] = None) -> IHyperOpt: """ Search and loads the specified hyperopt. :param hyperopt_name: name of the module to import + :param config: configuration dictionary :param extra_dir: additional directory to search for the given hyperopt :return: HyperOpt instance or None """ current_path = Path(__file__).parent.parent.joinpath('optimize').resolve() abs_paths = [ - Path.cwd().joinpath('user_data/hyperopts'), + config['user_data_dir'].joinpath('hyperopts'), current_path, ] @@ -79,7 +81,7 @@ class HyperOptLossResolver(IResolver): __slots__ = ['hyperoptloss'] - def __init__(self, config: Optional[Dict] = None) -> None: + def __init__(self, config: Dict = None) -> None: """ Load the custom class from config parameter :param config: configuration dictionary or None @@ -89,7 +91,7 @@ class HyperOptLossResolver(IResolver): # Verify the hyperopt is in the configuration, otherwise fallback to the default hyperopt hyperopt_name = config.get('hyperopt_loss') or DEFAULT_HYPEROPT_LOSS self.hyperoptloss = self._load_hyperoptloss( - hyperopt_name, extra_dir=config.get('hyperopt_path')) + hyperopt_name, config, extra_dir=config.get('hyperopt_path')) # Assign ticker_interval to be used in hyperopt self.hyperoptloss.__class__.ticker_interval = str(config['ticker_interval']) @@ -99,17 +101,19 @@ class HyperOptLossResolver(IResolver): f"Found hyperopt {hyperopt_name} does not implement `hyperopt_loss_function`.") def _load_hyperoptloss( - self, hyper_loss_name: str, extra_dir: Optional[str] = None) -> IHyperOptLoss: + self, hyper_loss_name: str, config: Dict, + extra_dir: Optional[str] = None) -> IHyperOptLoss: """ Search and loads the specified hyperopt loss class. :param hyper_loss_name: name of the module to import + :param config: configuration dictionary :param extra_dir: additional directory to search for the given hyperopt :return: HyperOptLoss instance or None """ current_path = Path(__file__).parent.parent.joinpath('optimize').resolve() abs_paths = [ - Path.cwd().joinpath('user_data/hyperopts'), + config['user_data_dir'].joinpath('hyperopts'), current_path, ] diff --git a/freqtrade/resolvers/iresolver.py b/freqtrade/resolvers/iresolver.py index 310c54015..8d93af568 100644 --- a/freqtrade/resolvers/iresolver.py +++ b/freqtrade/resolvers/iresolver.py @@ -58,7 +58,7 @@ class IResolver(object): if not str(entry).endswith('.py'): logger.debug('Ignoring %s', entry) continue - module_path = Path.resolve(directory.joinpath(entry)) + module_path = entry.resolve() obj = IResolver._get_valid_object( object_type, module_path, object_name ) diff --git a/freqtrade/resolvers/pairlist_resolver.py b/freqtrade/resolvers/pairlist_resolver.py index 3d95c0295..f38253155 100644 --- a/freqtrade/resolvers/pairlist_resolver.py +++ b/freqtrade/resolvers/pairlist_resolver.py @@ -25,21 +25,22 @@ class PairListResolver(IResolver): Load the custom class from config parameter :param config: configuration dictionary or None """ - self.pairlist = self._load_pairlist(pairlist_name, kwargs={'freqtrade': freqtrade, - 'config': config}) + self.pairlist = self._load_pairlist(pairlist_name, config, kwargs={'freqtrade': freqtrade, + 'config': config}) def _load_pairlist( - self, pairlist_name: str, kwargs: dict) -> IPairList: + self, pairlist_name: str, config: dict, kwargs: dict) -> IPairList: """ Search and loads the specified pairlist. :param pairlist_name: name of the module to import + :param config: configuration dictionary :param extra_dir: additional directory to search for the given pairlist :return: PairList instance or None """ current_path = Path(__file__).parent.parent.joinpath('pairlist').resolve() abs_paths = [ - Path.cwd().joinpath('user_data/pairlist'), + config['user_data_dir'].joinpath('pairlist'), current_path, ] diff --git a/freqtrade/resolvers/strategy_resolver.py b/freqtrade/resolvers/strategy_resolver.py index aa73327ff..37aa96b68 100644 --- a/freqtrade/resolvers/strategy_resolver.py +++ b/freqtrade/resolvers/strategy_resolver.py @@ -123,7 +123,7 @@ class StrategyResolver(IResolver): current_path = Path(__file__).parent.parent.joinpath('strategy').resolve() abs_paths = [ - Path.cwd().joinpath('user_data/strategies'), + config['user_data_dir'].joinpath('strategies'), current_path, ] diff --git a/freqtrade/tests/conftest.py b/freqtrade/tests/conftest.py index df9513cdb..654b959ca 100644 --- a/freqtrade/tests/conftest.py +++ b/freqtrade/tests/conftest.py @@ -239,6 +239,7 @@ def default_conf(): }, "initial_state": "running", "db_url": "sqlite://", + "user_data_dir": Path("user_data"), "verbosity": 3, } return configuration diff --git a/freqtrade/tests/optimize/test_hyperopt.py b/freqtrade/tests/optimize/test_hyperopt.py index 1c4e2445c..55ac6a739 100644 --- a/freqtrade/tests/optimize/test_hyperopt.py +++ b/freqtrade/tests/optimize/test_hyperopt.py @@ -1,12 +1,13 @@ # pragma pylint: disable=missing-docstring,W0212,C0103 import os from datetime import datetime -from unittest.mock import MagicMock +from unittest.mock import MagicMock, PropertyMock import pandas as pd import pytest from arrow import Arrow from filelock import Timeout +from pathlib import Path from freqtrade import DependencyException, OperationalException from freqtrade.data.converter import parse_ticker_dataframe @@ -14,8 +15,7 @@ from freqtrade.data.history import load_tickerdata_file from freqtrade.optimize import setup_configuration, start_hyperopt from freqtrade.optimize.default_hyperopt import DefaultHyperOpts from freqtrade.optimize.default_hyperopt_loss import DefaultHyperOptLoss -from freqtrade.optimize.hyperopt import (HYPEROPT_LOCKFILE, TICKERDATA_PICKLE, - Hyperopt) +from freqtrade.optimize.hyperopt import Hyperopt from freqtrade.resolvers.hyperopt_resolver import HyperOptResolver, HyperOptLossResolver from freqtrade.state import RunMode from freqtrade.strategy.interface import SellType @@ -54,11 +54,14 @@ def create_trials(mocker, hyperopt) -> None: - we might have a pickle'd file so make sure that we return false when looking for it """ - hyperopt.trials_file = os.path.join('freqtrade', 'tests', 'optimize', 'ut_trials.pickle') + hyperopt.trials_file = Path('freqtrade/tests/optimize/ut_trials.pickle') - mocker.patch('freqtrade.optimize.hyperopt.os.path.exists', return_value=False) - mocker.patch('freqtrade.optimize.hyperopt.os.path.getsize', return_value=1) - mocker.patch('freqtrade.optimize.hyperopt.os.remove', return_value=True) + mocker.patch.object(Path, "is_file", MagicMock(return_value=False)) + stat_mock = MagicMock() + stat_mock.st_size = PropertyMock(return_value=1) + mocker.patch.object(Path, "stat", MagicMock(return_value=False)) + + mocker.patch.object(Path, "unlink", MagicMock(return_value=True)) mocker.patch('freqtrade.optimize.hyperopt.dump', return_value=None) return [{'loss': 1, 'result': 'foo', 'params': {}}] @@ -264,7 +267,7 @@ def test_start_failure(mocker, default_conf, caplog) -> None: def test_start_filelock(mocker, default_conf, caplog) -> None: - start_mock = MagicMock(side_effect=Timeout(HYPEROPT_LOCKFILE)) + start_mock = MagicMock(side_effect=Timeout(Hyperopt.get_lock_filename(default_conf))) patched_configuration_load_config_file(mocker, default_conf) mocker.patch('freqtrade.optimize.hyperopt.Hyperopt.start', start_mock) patch_exchange(mocker) @@ -597,10 +600,10 @@ def test_clean_hyperopt(mocker, default_conf, caplog): }) mocker.patch("freqtrade.optimize.hyperopt.Path.is_file", MagicMock(return_value=True)) unlinkmock = mocker.patch("freqtrade.optimize.hyperopt.Path.unlink", MagicMock()) - Hyperopt(default_conf) + h = Hyperopt(default_conf) assert unlinkmock.call_count == 2 - assert log_has(f"Removing `{TICKERDATA_PICKLE}`.", caplog) + assert log_has(f"Removing `{h.tickerdata_pickle}`.", caplog) def test_continue_hyperopt(mocker, default_conf, caplog): diff --git a/freqtrade/tests/strategy/test_strategy.py b/freqtrade/tests/strategy/test_strategy.py index 6c2f6f78b..240b83b8b 100644 --- a/freqtrade/tests/strategy/test_strategy.py +++ b/freqtrade/tests/strategy/test_strategy.py @@ -60,62 +60,65 @@ def test_search_strategy(): assert s is None -def test_load_strategy(result): - resolver = StrategyResolver({'strategy': 'TestStrategy'}) +def test_load_strategy(default_conf, result): + default_conf.update({'strategy': 'TestStrategy'}) + resolver = StrategyResolver(default_conf) assert 'adx' in resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'}) -def test_load_strategy_base64(result, caplog): +def test_load_strategy_base64(result, caplog, default_conf): with open("user_data/strategies/test_strategy.py", "rb") as file: encoded_string = urlsafe_b64encode(file.read()).decode("utf-8") - resolver = StrategyResolver({'strategy': 'TestStrategy:{}'.format(encoded_string)}) + default_conf.update({'strategy': 'TestStrategy:{}'.format(encoded_string)}) + + resolver = StrategyResolver(default_conf) assert 'adx' in resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'}) # Make sure strategy was loaded from base64 (using temp directory)!! assert log_has_re(r"Using resolved strategy TestStrategy from '" + tempfile.gettempdir() + r"/.*/TestStrategy\.py'\.\.\.", caplog) -def test_load_strategy_invalid_directory(result, caplog): - resolver = StrategyResolver() +def test_load_strategy_invalid_directory(result, caplog, default_conf): + resolver = StrategyResolver(default_conf) extra_dir = Path.cwd() / 'some/path' - resolver._load_strategy('TestStrategy', config={}, extra_dir=extra_dir) + resolver._load_strategy('TestStrategy', config=default_conf, extra_dir=extra_dir) assert log_has_re(r'Path .*' + r'some.*path.*' + r'.* does not exist', caplog) assert 'adx' in resolver.strategy.advise_indicators(result, {'pair': 'ETH/BTC'}) -def test_load_not_found_strategy(): - strategy = StrategyResolver() +def test_load_not_found_strategy(default_conf): + strategy = StrategyResolver(default_conf) with pytest.raises(OperationalException, match=r"Impossible to load Strategy 'NotFoundStrategy'. " r"This class does not exist or contains Python code errors."): - strategy._load_strategy(strategy_name='NotFoundStrategy', config={}) + strategy._load_strategy(strategy_name='NotFoundStrategy', config=default_conf) -def test_load_staticmethod_importerror(mocker, caplog): +def test_load_staticmethod_importerror(mocker, caplog, default_conf): mocker.patch("freqtrade.resolvers.strategy_resolver.import_strategy", Mock( side_effect=TypeError("can't pickle staticmethod objects"))) with pytest.raises(OperationalException, match=r"Impossible to load Strategy 'DefaultStrategy'. " r"This class does not exist or contains Python code errors."): - StrategyResolver() + StrategyResolver(default_conf) assert log_has_re(r".*Error: can't pickle staticmethod objects", caplog) -def test_strategy(result): - config = {'strategy': 'DefaultStrategy'} +def test_strategy(result, default_conf): + default_conf.update({'strategy': 'DefaultStrategy'}) - resolver = StrategyResolver(config) + resolver = StrategyResolver(default_conf) metadata = {'pair': 'ETH/BTC'} assert resolver.strategy.minimal_roi[0] == 0.04 - assert config["minimal_roi"]['0'] == 0.04 + assert default_conf["minimal_roi"]['0'] == 0.04 assert resolver.strategy.stoploss == -0.10 - assert config['stoploss'] == -0.10 + assert default_conf['stoploss'] == -0.10 assert resolver.strategy.ticker_interval == '5m' - assert config['ticker_interval'] == '5m' + assert default_conf['ticker_interval'] == '5m' df_indicators = resolver.strategy.advise_indicators(result, metadata=metadata) assert 'adx' in df_indicators @@ -127,54 +130,54 @@ def test_strategy(result): assert 'sell' in dataframe.columns -def test_strategy_override_minimal_roi(caplog): +def test_strategy_override_minimal_roi(caplog, default_conf): caplog.set_level(logging.INFO) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'minimal_roi': { "0": 0.5 } - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.minimal_roi[0] == 0.5 assert log_has("Override strategy 'minimal_roi' with value in config file: {'0': 0.5}.", caplog) -def test_strategy_override_stoploss(caplog): +def test_strategy_override_stoploss(caplog, default_conf): caplog.set_level(logging.INFO) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'stoploss': -0.5 - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.stoploss == -0.5 assert log_has("Override strategy 'stoploss' with value in config file: -0.5.", caplog) -def test_strategy_override_trailing_stop(caplog): +def test_strategy_override_trailing_stop(caplog, default_conf): caplog.set_level(logging.INFO) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'trailing_stop': True - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.trailing_stop assert isinstance(resolver.strategy.trailing_stop, bool) assert log_has("Override strategy 'trailing_stop' with value in config file: True.", caplog) -def test_strategy_override_trailing_stop_positive(caplog): +def test_strategy_override_trailing_stop_positive(caplog, default_conf): caplog.set_level(logging.INFO) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'trailing_stop_positive': -0.1, 'trailing_stop_positive_offset': -0.2 - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.trailing_stop_positive == -0.1 assert log_has("Override strategy 'trailing_stop_positive' with value in config file: -0.1.", @@ -185,15 +188,15 @@ def test_strategy_override_trailing_stop_positive(caplog): caplog) -def test_strategy_override_ticker_interval(caplog): +def test_strategy_override_ticker_interval(caplog, default_conf): caplog.set_level(logging.INFO) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'ticker_interval': 60, 'stake_currency': 'ETH' - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.ticker_interval == 60 assert resolver.strategy.stake_currency == 'ETH' @@ -201,21 +204,21 @@ def test_strategy_override_ticker_interval(caplog): caplog) -def test_strategy_override_process_only_new_candles(caplog): +def test_strategy_override_process_only_new_candles(caplog, default_conf): caplog.set_level(logging.INFO) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'process_only_new_candles': True - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.process_only_new_candles assert log_has("Override strategy 'process_only_new_candles' with value in config file: True.", caplog) -def test_strategy_override_order_types(caplog): +def test_strategy_override_order_types(caplog, default_conf): caplog.set_level(logging.INFO) order_types = { @@ -224,12 +227,11 @@ def test_strategy_override_order_types(caplog): 'stoploss': 'limit', 'stoploss_on_exchange': True, } - - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'order_types': order_types - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.order_types for method in ['buy', 'sell', 'stoploss', 'stoploss_on_exchange']: @@ -239,18 +241,18 @@ def test_strategy_override_order_types(caplog): " {'buy': 'market', 'sell': 'limit', 'stoploss': 'limit'," " 'stoploss_on_exchange': True}.", caplog) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'order_types': {'buy': 'market'} - } + }) # Raise error for invalid configuration with pytest.raises(ImportError, match=r"Impossible to load Strategy 'DefaultStrategy'. " r"Order-types mapping is incomplete."): - StrategyResolver(config) + StrategyResolver(default_conf) -def test_strategy_override_order_tif(caplog): +def test_strategy_override_order_tif(caplog, default_conf): caplog.set_level(logging.INFO) order_time_in_force = { @@ -258,11 +260,11 @@ def test_strategy_override_order_tif(caplog): 'sell': 'gtc', } - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'order_time_in_force': order_time_in_force - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.order_time_in_force for method in ['buy', 'sell']: @@ -271,61 +273,61 @@ def test_strategy_override_order_tif(caplog): assert log_has("Override strategy 'order_time_in_force' with value in config file:" " {'buy': 'fok', 'sell': 'gtc'}.", caplog) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'order_time_in_force': {'buy': 'fok'} - } + }) # Raise error for invalid configuration with pytest.raises(ImportError, match=r"Impossible to load Strategy 'DefaultStrategy'. " r"Order-time-in-force mapping is incomplete."): - StrategyResolver(config) + StrategyResolver(default_conf) -def test_strategy_override_use_sell_signal(caplog): +def test_strategy_override_use_sell_signal(caplog, default_conf): caplog.set_level(logging.INFO) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert not resolver.strategy.use_sell_signal assert isinstance(resolver.strategy.use_sell_signal, bool) # must be inserted to configuration - assert 'use_sell_signal' in config['experimental'] - assert not config['experimental']['use_sell_signal'] + assert 'use_sell_signal' in default_conf['experimental'] + assert not default_conf['experimental']['use_sell_signal'] - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'experimental': { 'use_sell_signal': True, }, - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.use_sell_signal assert isinstance(resolver.strategy.use_sell_signal, bool) assert log_has("Override strategy 'use_sell_signal' with value in config file: True.", caplog) -def test_strategy_override_use_sell_profit_only(caplog): +def test_strategy_override_use_sell_profit_only(caplog, default_conf): caplog.set_level(logging.INFO) - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert not resolver.strategy.sell_profit_only assert isinstance(resolver.strategy.sell_profit_only, bool) # must be inserted to configuration - assert 'sell_profit_only' in config['experimental'] - assert not config['experimental']['sell_profit_only'] + assert 'sell_profit_only' in default_conf['experimental'] + assert not default_conf['experimental']['sell_profit_only'] - config = { + default_conf.update({ 'strategy': 'DefaultStrategy', 'experimental': { 'sell_profit_only': True, }, - } - resolver = StrategyResolver(config) + }) + resolver = StrategyResolver(default_conf) assert resolver.strategy.sell_profit_only assert isinstance(resolver.strategy.sell_profit_only, bool) @@ -333,10 +335,11 @@ def test_strategy_override_use_sell_profit_only(caplog): @pytest.mark.filterwarnings("ignore:deprecated") -def test_deprecate_populate_indicators(result): +def test_deprecate_populate_indicators(result, default_conf): default_location = path.join(path.dirname(path.realpath(__file__))) - resolver = StrategyResolver({'strategy': 'TestStrategyLegacy', - 'strategy_path': default_location}) + default_conf.update({'strategy': 'TestStrategyLegacy', + 'strategy_path': default_location}) + resolver = StrategyResolver(default_conf) with warnings.catch_warnings(record=True) as w: # Cause all warnings to always be triggered. warnings.simplefilter("always") @@ -366,10 +369,11 @@ def test_deprecate_populate_indicators(result): @pytest.mark.filterwarnings("ignore:deprecated") -def test_call_deprecated_function(result, monkeypatch): +def test_call_deprecated_function(result, monkeypatch, default_conf): default_location = path.join(path.dirname(path.realpath(__file__))) - resolver = StrategyResolver({'strategy': 'TestStrategyLegacy', - 'strategy_path': default_location}) + default_conf.update({'strategy': 'TestStrategyLegacy', + 'strategy_path': default_location}) + resolver = StrategyResolver(default_conf) metadata = {'pair': 'ETH/BTC'} # Make sure we are using a legacy function diff --git a/freqtrade/tests/test_configuration.py b/freqtrade/tests/test_configuration.py index fc3adad44..6d74f8b37 100644 --- a/freqtrade/tests/test_configuration.py +++ b/freqtrade/tests/test_configuration.py @@ -13,7 +13,8 @@ from freqtrade import OperationalException, constants from freqtrade.configuration import Arguments, Configuration, validate_config_consistency from freqtrade.configuration.check_exchange import check_exchange from freqtrade.configuration.config_validation import validate_config_schema -from freqtrade.configuration.create_datadir import create_datadir +from freqtrade.configuration.directory_operations import (create_datadir, + create_userdata_dir) from freqtrade.configuration.load_config import load_config_file from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL from freqtrade.loggers import _set_loggers @@ -52,6 +53,7 @@ def test_load_config_incorrect_stake_amount(default_conf) -> None: def test_load_config_file(default_conf, mocker, caplog) -> None: + del default_conf['user_data_dir'] file_mock = mocker.patch('freqtrade.configuration.load_config.open', mocker.mock_open( read_data=json.dumps(default_conf) )) @@ -331,6 +333,7 @@ def test_setup_configuration_without_arguments(mocker, default_conf, caplog) -> assert 'exchange' in config assert 'pair_whitelist' in config['exchange'] assert 'datadir' in config + assert 'user_data_dir' in config assert log_has('Using data directory: {} ...'.format(config['datadir']), caplog) assert 'ticker_interval' in config assert not log_has('Parameter -i/--ticker-interval detected ...', caplog) @@ -355,11 +358,15 @@ def test_setup_configuration_with_arguments(mocker, default_conf, caplog) -> Non 'freqtrade.configuration.configuration.create_datadir', lambda c, x: x ) - + mocker.patch( + 'freqtrade.configuration.configuration.create_userdata_dir', + lambda x, *args, **kwargs: Path(x) + ) arglist = [ '--config', 'config.json', '--strategy', 'DefaultStrategy', '--datadir', '/foo/bar', + '--userdir', "/tmp/freqtrade", 'backtesting', '--ticker-interval', '1m', '--live', @@ -380,7 +387,10 @@ def test_setup_configuration_with_arguments(mocker, default_conf, caplog) -> Non assert 'exchange' in config assert 'pair_whitelist' in config['exchange'] assert 'datadir' in config - assert log_has('Using data directory: {} ...'.format(config['datadir']), caplog) + assert log_has('Using data directory: {} ...'.format("/foo/bar"), caplog) + assert log_has('Using user-data directory: {} ...'.format("/tmp/freqtrade"), caplog) + assert 'user_data_dir' in config + assert 'ticker_interval' in config assert log_has('Parameter -i/--ticker-interval detected ... Using ticker_interval: 1m ...', caplog) @@ -615,6 +625,35 @@ def test_create_datadir(mocker, default_conf, caplog) -> None: assert log_has('Created data directory: /foo/bar', caplog) +def test_create_userdata_dir(mocker, default_conf, caplog) -> None: + mocker.patch.object(Path, "is_dir", MagicMock(return_value=False)) + md = mocker.patch.object(Path, 'mkdir', MagicMock()) + + x = create_userdata_dir('/tmp/bar', create_dir=True) + assert md.call_count == 7 + assert md.call_args[1]['parents'] is False + assert log_has('Created user-data directory: /tmp/bar', caplog) + assert isinstance(x, Path) + assert str(x) == "/tmp/bar" + + +def test_create_userdata_dir_exists(mocker, default_conf, caplog) -> None: + mocker.patch.object(Path, "is_dir", MagicMock(return_value=True)) + md = mocker.patch.object(Path, 'mkdir', MagicMock()) + + create_userdata_dir('/tmp/bar') + assert md.call_count == 0 + + +def test_create_userdata_dir_exists_exception(mocker, default_conf, caplog) -> None: + mocker.patch.object(Path, "is_dir", MagicMock(return_value=False)) + md = mocker.patch.object(Path, 'mkdir', MagicMock()) + + with pytest.raises(OperationalException, match=r'Directory `/tmp/bar` does not exist.*'): + create_userdata_dir('/tmp/bar', create_dir=False) + assert md.call_count == 0 + + def test_validate_tsl(default_conf): default_conf['trailing_stop'] = True default_conf['trailing_stop_positive'] = 0 diff --git a/freqtrade/tests/test_plotting.py b/freqtrade/tests/test_plotting.py index 94d40ab84..a78e38c1f 100644 --- a/freqtrade/tests/test_plotting.py +++ b/freqtrade/tests/test_plotting.py @@ -1,5 +1,6 @@ from copy import deepcopy +from pathlib import Path from unittest.mock import MagicMock import plotly.graph_objects as go @@ -209,7 +210,8 @@ def test_generate_Plot_filename(): def test_generate_plot_file(mocker, caplog): fig = generage_empty_figure() plot_mock = mocker.patch("freqtrade.plot.plotting.plot", MagicMock()) - store_plot_file(fig, filename="freqtrade-plot-UNITTEST_BTC-5m.html") + store_plot_file(fig, filename="freqtrade-plot-UNITTEST_BTC-5m.html", + directory=Path("user_data/plots")) assert plot_mock.call_count == 1 assert plot_mock.call_args[0][0] == fig diff --git a/freqtrade/tests/test_utils.py b/freqtrade/tests/test_utils.py index 003b3b286..d04e62b28 100644 --- a/freqtrade/tests/test_utils.py +++ b/freqtrade/tests/test_utils.py @@ -6,8 +6,8 @@ import pytest from freqtrade.state import RunMode from freqtrade.tests.conftest import get_args, log_has, patch_exchange -from freqtrade.utils import (setup_utils_configuration, start_download_data, - start_list_exchanges) +from freqtrade.utils import (setup_utils_configuration, start_create_userdir, + start_download_data, start_list_exchanges) def test_setup_utils_configuration(): @@ -47,6 +47,29 @@ def test_list_exchanges(capsys): assert re.search(r"^bittrex$", captured.out, re.MULTILINE) +def test_create_datadir_failed(caplog): + + args = [ + "create-userdir", + ] + with pytest.raises(SystemExit): + start_create_userdir(get_args(args)) + assert log_has("`create-userdir` requires --userdir to be set.", caplog) + + +def test_create_datadir(caplog, mocker): + cud = mocker.patch("freqtrade.utils.create_userdata_dir", MagicMock()) + args = [ + "create-userdir", + "--userdir", + "/temp/freqtrade/test" + ] + start_create_userdir(get_args(args)) + + assert cud.call_count == 1 + assert len(caplog.record_tuples) == 0 + + def test_download_data(mocker, markets, caplog): dl_mock = mocker.patch('freqtrade.utils.download_pair_history', MagicMock()) patch_exchange(mocker) diff --git a/freqtrade/utils.py b/freqtrade/utils.py index 002d89738..56e60ec82 100644 --- a/freqtrade/utils.py +++ b/freqtrade/utils.py @@ -7,6 +7,7 @@ from typing import Any, Dict import arrow from freqtrade.configuration import Configuration, TimeRange +from freqtrade.configuration.directory_operations import create_userdata_dir from freqtrade.data.history import download_pair_history from freqtrade.exchange import available_exchanges from freqtrade.resolvers import ExchangeResolver @@ -46,6 +47,19 @@ def start_list_exchanges(args: Namespace) -> None: f"{', '.join(available_exchanges())}") +def start_create_userdir(args: Namespace) -> None: + """ + Create "user_data" directory to contain user data strategies, hyperopts, ...) + :param args: Cli args from Arguments() + :return: None + """ + if "user_data_dir" in args and args.user_data_dir: + create_userdata_dir(args.user_data_dir, create_dir=True) + else: + logger.warning("`create-userdir` requires --userdir to be set.") + sys.exit(1) + + def start_download_data(args: Namespace) -> None: """ Download data (former download_backtest_data.py script) diff --git a/scripts/plot_dataframe.py b/scripts/plot_dataframe.py index 6412c45a4..db4f99d61 100755 --- a/scripts/plot_dataframe.py +++ b/scripts/plot_dataframe.py @@ -63,7 +63,8 @@ def analyse_and_plot_pairs(config: Dict[str, Any]): indicators2=config["indicators2"].split(",") ) - store_plot_file(fig, generate_plot_filename(pair, config['ticker_interval'])) + store_plot_file(fig, filename=generate_plot_filename(pair, config['ticker_interval']), + directory=config['user_data_dir'] / "plot") logger.info('End of ploting process %s plots generated', pair_counter) diff --git a/scripts/plot_profit.py b/scripts/plot_profit.py index 4290bca45..578ddf15f 100755 --- a/scripts/plot_profit.py +++ b/scripts/plot_profit.py @@ -32,7 +32,8 @@ def plot_profit(config: Dict[str, Any]) -> None: # Create an average close price of all the pairs that were involved. # this could be useful to gauge the overall market trend fig = generate_profit_graph(plot_elements["pairs"], plot_elements["tickers"], trades) - store_plot_file(fig, filename='freqtrade-profit-plot.html', auto_open=True) + store_plot_file(fig, filename='freqtrade-profit-plot.html', + directory=config['user_data_dir'] / "plot", auto_open=True) def plot_parse_args(args: List[str]) -> Dict[str, Any]: diff --git a/user_data/backtest_results/.gitkeep b/user_data/backtest_results/.gitkeep new file mode 100644 index 000000000..e69de29bb