Deep-learning-based stock market prediction incorporating ESG sentiment and technical indicators

This section describes the experimental flow. First, data were collected for the experiment. Subsequently, preprocessing was performed to eliminate irrelevant textual data. Third, technical indicators were derived from the S&P 500 dataset, with sentiment scores generated from ESG-related news data. After combining the processed data, the scaled data were adjusted as input data for the deep learning models to forecast future prices. Lastly, MAPE was employed as the assessment measure for regression performance. In addition, ablation tests were performed to evaluate the effectiveness of each input feature. The experimental procedure is illustrated in Fig. 1.

Flowchart for predicting S&P 500 index.

Data collection

The S&P 500 index is used to grasp and monitor the overall trends of the stock market and is considered one of the indicators representing the health of the United States’ financial markets²⁶. The S&P 500 represents an index 500 major U.S. companies, it reflects market-wide movements rather than individual company stock prices. In addition, the S&P 500 includes companies from a variety of industries and sectors. Therefore, constructing a stock price prediction model including data from various industries is equivalent to designing a generalized model with versatility. Moreover, while stocks of individual companies must also consider the influence of internal factors, the S&P 500 is influenced by the overall market perception²⁷. Consequently, building an enhanced stock price prediction model by integrating ESG information and the S&P 500 can underscore the significance and impact of sustainability information across the market to investors and relevant researchers.

The experiments were conducted by gathering two datasets spanning from January 1, 2016, to July 31, 2023. Through LexisNexis, the authors accessed and collected a collection of 14,049 news articles using the search term “ESG.” Access to the LexisNexis database may require a paid subscription, such as institutional access. Additionally, historical data on the S&P 500 index, containing information such as date, closing value, opening value, high value, low value, trading volume, and volatility, for the same time periods were sourced from investing.com.

Feature engineering

Based on previous research, the authors obtained various technical indicators that have been shown to impact stock prices using the TA-lib module^28,29. The chosen features were opening price, closing price, high price, low price, trading volume, RSI, SMA_5, SMA_20, EMA, MACD, signal, Stochastic RSI_fastk, Stochastic RSI_fastd, Stochastic Oscillator Index_slowk, Stochastic Oscillator Index_slowd, stochastic oscillator index_slowd, WilliamR, Momentum, and ROC. Detailed descriptions of these technical indicators are provided below.

The opening price is the price of a stock at the beginning of a trading session and indicates the first transaction made for the day. High prices represent the highest value of a stock trade within a specific trading period, whereas low prices signify the lowest. Trading volume, which reflects market activity, is the number of shares or contracts traded during a specific period.

The RSI is a momentum oscillator that measures the speed and change in price movements and helps identify overbought or oversold conditions. SMAs are average closing prices over a specified number of periods. For instance, SMA_5 and SMA_20 represent the 5-day and 20-day moving averages, respectively. The EMA responds better to recent price changes by assigning more weight to them³⁰.

MACD is a momentum indicator that follows trends by illustrating the interaction between two moving averages of a security’s price. Signal lines, i.e., the moving averages derived from MACD lines, play an important role in generating valuable buy-and-sell signals for traders and investors³¹.

Stochastic RSI_fastk and Stochastic RSI_fastd computed based on both the RSI and stochastic oscillator effectively grasp potential points of price reversal and enhance the accuracy of predictions³². To ensure smoothness, the stochastic oscillator indices_slowk and stochastic oscillator indices_slowd were considered supplementary components of the stochastic oscillator.

Another integral aspect of the analysis was William’s %R, commonly referred to as Williams R. This momentum indicator assesses whether market conditions indicate overbought or oversold scenarios, thereby contributing to a comprehensive understanding of market sentiment³³.

Next indicators employed is momentum. The concept of momentum can be used to measure the rate of price change. Momentum provides insights into the rate at which prices change by quantifying the rate of change in stock prices. Finally, the ROC, a metric similar to momentum, involves calculating changes in prices over a specific period, providing insights into the extent of price fluctuations³⁴.

Sentiment index calculation using financial bidirectional encoder representations from transformers (FinBERT)

Preprocessing including stopwords removal and lemmatization was conducted on the news data, followed by sentiment analysis using FinBERT. FinBERT is built upon the BERT architecture, which is an effective language model for natural language processing and understanding by encoding text by considering context bidirectionally³⁵. FinBERT specializes in domain knowledge by retraining BERT’s pretrained model with financial data. FinBERT takes financial-related texts such as financial news, reports, and web posts as inputs, and analyses and predicts the sentiment of the text, categorizing it as either positive, negative, or neutral.

The scores in the data were labeled 0 for negative sentiments and 1 for positive sentiments (Eq. (1)). Referring to a study by Wu et al.³⁶, sentiment measurements were calculated as the difference between the number of negative and positive posts in a specific dataset.

where \({M}_{tpos}\) represents the number of positive news articles and \({M}_{tneg}\) represents the number of negative articles on day t. The range of values for the sentiment index was between −1 and 1²⁵. If the sentiment index value approaches −1, it suggests a negative tone in the news for that date. Conversely, if it approaches 1, it indicates an overall positive tone in the news. Before employing the selected features as input to the framework, a min–max scaler was applied to standardize the range of these values between 0 and 1.

Window size

Subsequently, multiple datasets are generated, each corresponding to a distinct hyperparameter window. Window size is a fundamental concept in stock price predictions for processing and predicting time-series data^37,38. The window size defines a fixed unit period, with the data within this window used to predict future stock prices. Therefore, selecting an appropriate window size is crucial to improving the performance of stock price prediction models. In this study, experiments were conducted using three window sizes: 3, 4, and 5 (Fig. 2). Finally, the training and test datasets were split at an 8:2 ratio. The validation dataset comprises 20% of the training dataset.

Window size illustration.

Deep learning models

Bidirectional recurrent neural networks (Bi-RNN) are a type of recurrent neural network capable of considering both the preceding and subsequent contexts of a sequence. This bidirectional characteristic enables them to capture patterns in different temporal directions³⁹. Moreover, since short-term factors can influence the fluctuation in stock prices, the RNN structure with recurrent layers is adept at capturing these changes, rendering it suitable for application as a time series model. Additionally, Bi-RNN has a flexible structure that can be applied to various types of time series data, making it useful for processing patterns. By contrast, bidirectional long short-term memory networks (Bi-LSTM) represent an enhanced iteration of RNNs that incorporate LSTM cells⁴⁰. They excel at learning long-range dependencies and are particularly effective in tasks involving sequential data, such as time-series forecasting⁴¹.