A multiagent approach to Q-learning for daily stock trading
The portfolio management for trading in the stock market poses a challenging stochastic control problem of significant commercial interests to finance industry. To date, many researchers have proposed various methods to build an intelligent portfolio management system that can recommend financial decisions for daily stock trading. Many promising results have been reported from the supervised learning community on the possibility of building a profitable trading system. More recently, several studies have shown that even the problem of integrating stock price prediction results with trading strategies can be successfully addressed by applying reinforcement learning algorithms. Motivated by this, we present a new stock trading framework that attempts to further enhance the performance of reinforcement learning-based systems. The proposed approach incorporates multiple Q-learning agents, allowing them to effectively divide and conquer the stock trading problem by defining necessary roles for cooperatively carrying out stock pricing and selection decisions. Furthermore, in an attempt to address the complexity issue when considering a large amount of data to obtain long-term dependence among the stock prices, we present a representation scheme that can succinctly summarize the history of price changes. Experimental results on a Korean stock market show that the proposed trading framework outperforms those trained by other alternative approaches both in terms of profit and risk management.
Related papers
Stock trading is an important decision-making problem that involves both stock selection and asset management. Though many promising results have been reported for predicting prices, selecting stocks, and managing assets using machine-learning techniques, considering all of them is challenging because of their complexity. In this paper, we present a new stock trading method that incorporates dynamic asset allocation in a reinforcement-learning framework. The proposed asset allocation strategy, called meta policy (MP), is designed to utilize the temporal information from both stock recommendations and the ratio of the stock fund over the asset. Local traders are constructed with pattern-based multiple predictors, and used to decide the purchase money per recommendation. Formulating the MP in the reinforcement learning framework is achieved by a compact design of the environment and the learning agent. Experimental results
Developing a successful strategy for stock trading is a vital task for investors. However, it is challenging to obtain an optimal strategy, given the complex and dynamic nature of the stock market. A reinforcement learning approach is applied because of its ability to learn from cumulative rewards and manage domains with stochastic transitions without a model. This thesis aims to explore RL’s applications to maximize returns from market investment, keeping in mind the human aspect of trading. The learnin agent trained on stock prices and public interest patterns data-mined from Google trends in the form of candlesticks graphs. Deep Q learning (DQN) has been used to train an agent based on fused images of stock data and Google trends data via a convolution neural network (CNN). This thesis explores the novel approach of using social media trends in the form of graphical information to augment training a reinforcement learning agent for stock trading. A study of different stocks paired with appropriate keywords was done to assess the effect of correlation between datasets on the agent’s ability to perform. Preliminary results show that the agent performs well when trained on correlated datasets getting cumulative positive rewards.
High-frequency trading (HFT) in financial markets requires sophisticated strategies to swiftly react to market dynamics and exploit profitable opportunities. This study aims to develop a novel trading strategy for highfrequency finance using reinforcement learning techniques, specifically Deep Q-Networks (DQN). The objective is to optimize trading decisions in real-time, leveraging historical market data and learning from past experiences to maximize profitability. Introduce a powerful and adaptable framework that can recognize intricate patterns and make the best trading decisions in dynamic market conditions by utilizing DQN. Experimental results demonstrate the effectiveness of the proposed Deep Q-Network approach in optimizing trading strategies for high-frequency finance. Our study highlights the potential of reinforcement learning, specifically Deep Q-Networks, in revolutionizing high-frequency trading strategies. Provide a framework that can learn on its own and optimize trading decisions by utilizing artificial intelligence. This will provide a competitive edge in the fast-paced financial markets of today.
Algorithm trading is a hot topic in machine learning. Reinforcement learning is widely used in financial trading because it can directly teach behavioural rules using rational rewards. A typical financial trading strategy application is flexible and expresses the state of multiple stocks. Limited trading activities for financial products were possible due to the difficulty of designing a flexible action space. Due to the inherent transformation of the market base, it is difficult to extract effective characteristics during trading battles from price fluctuations in financial markets, including perhaps noise, and models learned from past price data. It does not work well with unknown price data. Thus, we achieved effective feature extraction from raw price data by DNN, online reinforcement learning adaptation to unknown price factors, and flexible asset management across multiple stocks in this study. The Casual dilated convolution layer is a DNN layer architecture that can capture l.
Quantitative trading through automated systems has been vastly growing in recent years. The advancement in machine learning algorithms has pushed that growth even further, where their capability in extracting high-level patterns within financial markets data is evident. Nonetheless, trading with supervised machine learning can be challenging since the system learns to predict the price to minimize the error rather than optimize a financial performance measure. Reinforcement Learning (RL), a machine learning paradigm that intersects with optimal control theory, could bridge that divide since it is a goal-oriented learning system that could perform the two main trading steps, market analysis and making decisions to optimize a financial measure, without explicitly predicting the future price movement. This survey reviews quantitative trading under the different main RL methods. We first begin by describing the trading process and how it suits the RL framework, and we briefly discuss th.
arXiv (Cornell University), 2022
Financial domain tasks, such as trading in market exchanges, are challenging and have long attracted researchers. The recent achievements and the consequent notoriety of Reinforcement Learning (RL) have also increased its adoption in trading tasks. RL uses a framework with well-established formal concepts, which raises its attractiveness in learning profitable trading strategies. However, RL use without due attention in the financial area can prevent new researchers from following standards or failing to adopt relevant conceptual guidelines. In this work, we embrace the seminal RL technical fundamentals, concepts, and recommendations to perform a unified, theoreticallygrounded examination and comparison of previous research that could serve as a structuring guide for the field of study. A selection of twenty-nine articles was reviewed under our classification that considers RL’s most common formulations and design patterns from a large volume of available studies. This classification allowed for precise inspection of the most relevant aspects regarding data input, preprocessing, state and action composition, adopted RL techniques, evaluation setups, and overall results. Our analysis approach organized around fundamental RL concepts allowed for a clear identification of current system design best practices, gaps that require further investigation, and promising research opportunities. Finally, this review attempts to promote the development of this field of study by facilitating researchers’ commitment to standards adherence and helping them to avoid straying away from the RL constructs’ firm ground.
Presently, the volatile and dynamic aspects of stock prices are significant research challenges for stock markets or any other financial sector to design accurate and profitable trading strategies in all market situations. To meet such challenges, the usage of computer-aided stock trading techniques has grown in prominence in recent decades owing to their ability to rapidly and accurately analyze stock market situations. In the recent past, deep reinforcement learning (DRL) methods and trading bots are commonly utilized for algorithmic trading. However, in the existing literature, the trading agents employ the historical and present trends of stock prices as an observing state to make trading decisions without taking into account the long-term market future pattern of stock prices. Therefore, in this study, we proposed a novel decision support system for automated stock trading based on deep reinforcement learning that observes both past and future trends of stock prices whether single and multi-step ahead as an observing state to make the optimal trading decisions of buying, selling, and holding the stocks. More specifically, at every time step, future trends are monitored concurrently using a forecasting network whose output is concatenated with past trends of stock prices. The concatenated vectors are subsequently supplied to the DRL agent as an observation state. In addition, the suggested forecasting network is built on a Gated Recurrent Unit (GRU). The GRU-based agent captures more informative and inherent aspects of time-series financial data. Furthermore, the suggested decision support system has been tested on several stock markets such as Tesla, IBM, Amazon, CSCO, and Chinese Stocks as well as equity markets i-e SSE Composite Index, NIFTY 50 Index, US Commodity Index Fund, and has achieved encouraging profit values while trading. INDEX TERMS Decision support system, automated stock trading, deep reinforcement learning, deep-Q networks, forecasting network, GRU, long-term market future patterns.
Background and Objectives: Stock recommender system (SRS) based on deep reinforcement learning (DRL) has garnered significant attention within the financial research community. A robust DRL agent aims to consistently allocate some amount of cash to the combination of high-risk and low-risk stocks with the ultimate objective of maximizing returns and balancing risk. However, existing DRL-based SRSs focus on one or, at most, two sequential trading agents that operate within the same or shared environment, and often make mistakes in volatile or variable market conditions. In this paper, a robust Concurrent Multiagent Deep Reinforcement Learning-based Stock Recommender System (CMSRS) is proposed. Methods: The proposed system introduces a multi-layered architecture that includes feature extraction at the data layer to construct multiple trading environments, so that different feed DRL agents would robustly recommend assets for trading layer. The proposed CMSRS uses a variety of data sources, including Google stock trends, fundamental data and technical indicators along with historical price data, for the selection and recommendation suitable stocks to buy or sell concurrently by multiple agents. To optimize hyperparameters during the validation phase, we employ Sharpe ratio as a risk adjusted return measure. Additionally, we address liquidity requirements by defining a precise reward function that dynamically manages cash reserves. We also penalize the model for failing to maintain a reserve of cash. Results: The empirical results on the real U.S. stock market data show the superiority of our CMSRS, especially in volatile markets and out-of-sample data. Conclusion: The proposed CMSRS demonstrates significant advancements in stock recommendation by effectively leveraging multiple trading agents and diverse data sources. The empirical results underscore its robustness and superior performance, particularly in volatile market conditions. This multi-layered approach not only optimizes returns but also efficiently manages risks and liquidity, offering a compelling solution for dynamic and uncertain financial environments. Future work could further refine the model’s adaptability to other market conditions and explore its applicability across different asset classes.
Designing a profitable trading strategy plays a critical role in algorithmic trading, where the algorithm can manage and execute automated trading decisions. Determining a specific trading rule for trading at a particular time is a critical research problem in financial market trading. However, an intelligent, and a dynamic algorithmic trading driven by the current patterns of a given price time-series may help deal with this issue. Thus, Reinforcement Learning (RL) can achieve optimal dynamic algorithmic trading by considering the price time-series as its environment. A comprehensive representation of the environment states is indeed vital for proposing a dynamic algorithmic trading using RL. Therefore, we propose a representation of the environment states using the Directional Change (DC) event approach with a dynamic DC threshold. We refer to the proposed algorithmic trading approach as the DCRL trading strategy. In addition, the proposed DCRL trading strategy was trained using t.
International Journal of Grid and Distributed Computing, 2018
Recently, interest in financial transactions is increasing, and the number of investors in the stock market is increasing. These investors are applying financial analysis methods to stock trading in order to gain more profits, and combining with artificial intelligence techniques has made it possible to achieve returns in excess of the market average. As a result, the stock trading system based on reinforcement learning has attracted attention, and in recent years, studies are being conducted to optimize financial time series data by Multi-Agent Reinforcement Learning (MARL). However, MARL, which is used in existing stock trading, cannot be fully collaborated because of lack of generalization of experience. Therefore, in this paper, we propose Multi-agent Collaborated Network (MCN) that can share and generalize the experience by agent, and experiment on collaboration in distributed stock trading.
Title: Optimizing Trading Strategies in Quantitative Markets using Multi-Agent Reinforcement Learning
Abstract: Quantitative markets are characterized by swift dynamics and abundant uncertainties, making the pursuit of profit-driven stock trading actions inherently challenging. Within this context, reinforcement learning (RL), which operates on a reward-centric mechanism for optimal control, has surfaced as a potentially effective solution to the intricate financial decision-making conundrums presented. This paper delves into the fusion of two established financial trading strategies, namely the constant proportion portfolio insurance (CPPI) and the time-invariant portfolio protection (TIPP), with the multi-agent deep deterministic policy gradient (MADDPG) framework. As a result, we introduce two novel multi-agent RL (MARL) methods, CPPI-MADDPG and TIPP-MADDPG, tailored for probing strategic trading within quantitative markets. To validate these innovations, we implemented them on a diverse selection of 100 real-market shares. Our empirical findings reveal that the CPPI-MADDPG and TIPP-MADDPG strategies consistently outpace their traditional counterparts, affirming their efficacy in the realm of quantitative trading.
| Subjects: | Trading and Market Microstructure (q-fin.TR) ; Machine Learning (cs.LG); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2303.11959 [q-fin.TR] |
| (or arXiv:2303.11959v2 [q-fin.TR] for this version) | |
| https://doi.org/10.48550/arXiv.2303.11959 |
Focus to learn more
arXiv-issued DOI via DataCite
Submission history
From: Hengxi Zhang [view email]
[v1] Wed, 15 Mar 2023 11:47:57 UTC (709 KB)
[v2] Fri, 22 Dec 2023 04:59:00 UTC (709 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Optimizing Trading Strategies in Quantitative Markets using Multi-Agent Reinforcement Learning, by Hengxi Zhang and 5 other authors
https://www.academia.edu/55275402/A_multiagent_approach_to_Q_learning_for_daily_stock_tradinghttps://arxiv.org/abs/2303.11959
