Function process_data

data_processing/data_process.py:11–446

Function to pre-process LOBSTER data. The data must be stored in the input_path directory as 'daily message LOB' and 'orderbook' files. The data are treated in the following way: - Orderbook's states with crossed quotes are removed. - Each state in the orderbook is time-stamped, wi

(
        ticker: str,
        input_path: str,
        output_path: str,
        logs_path: str,
        horizons: list[int],
        normalization_window: int,
        time_index: str = "seconds",
        features: str = "orderbooks",
        scaling: bool = True,
)

def process_data(
        ticker: str,
        input_path: str,
        output_path: str,
        logs_path: str,
        horizons: list[int],
        normalization_window: int,
        time_index: str = "seconds",
        features: str = "orderbooks",
        scaling: bool = True,
) -> None:
    """
    Function to pre-process LOBSTER data. The data must be stored in the input_path directory as 'daily message LOB' and 'orderbook' files.

    The data are treated in the following way:
    - Orderbook's states with crossed quotes are removed.
    - Each state in the orderbook is time-stamped, with states occurring at the same time collapsed onto the last occurring state.
    - The first and last 10 minutes of market activity (inside usual opening times) are dropped.
    - Rolling z-score normalization is applied to the data, i.e. the mean and standard deviation of the previous 5 days is used to normalize current day's data.
      Hence, the first 5 days are dropped.
    - Smoothed returns at the requested horizons (in orderbook's changes) are returned:
       - if smoothing = "horizon": l = (m+ - m)/m, where m+ denotes the mean of the next h mid-prices, m(.) is current mid-price.
       - if smoothing = "uniform": l = (m+ - m)/m, where m+ denotes the mean of the k+1 mid-prices centered at m(. + h), m(.) is current mid-price.

    A log file is produced tracking:
    - Orderbook's files with problems.
    - Message orderbook's files with problems.
    - Trading days with unusual opening - closing times.
    - Trading days with crossed quotes.

    A statistics.csv file summarizes the following (daily) statistics:
    - # Updates (000): the total number of changes in the orderbook file.
    - # Trades (000): the total number of trades, computed by counting the number of message book events corresponding to the execution of (possibly hidden)
                      limit orders ('event_type' 4 or 5 in LOBSTER orderbook's message file).
    - # Price Changes (000): the total number of price changes per day.
    - # Price (USD): average price on the day, weighted average by time.
    - # Spread (bps): average spread on the day, weighted average by time.
    - # Volume (USD MM): total volume traded on the day, computed as the sum of the volumes of all the executed trades ('event_type' 4 or 5 in LOBSTER orderbook's message file).
                         The volume of a single trade is given by size*price.
    - # Tick size: the fraction of time that the bid-ask spread is equal to one tick for each stock.

    Args:
        ticker (str): The ticker to be considered.
        input_path (str): The path where the order book and message book files are stored, order book files have shape (:, 4*levels):
                       ["ASKp1", "ASKs1", "BIDp1",  "BIDs1", ..., "ASKp10", "ASKs10", "BIDp10",  "BIDs10"].
        output_path (str): The path where we wish to save the processed datasets.
        logs_path (str): The path where we wish to save the logs.
        time_index (str): The time-index to use ("seconds" or "datetime").
        horizons (list): Forecasting horizons for labels.
        normalization_window (int): Window for rolling z-score normalization.
        features (str): Whether to return 'orderbooks' or 'orderflows'.
        scaling (bool): Whether to apply rolling z-score normalization.

    Returns:
        None.
    """

    csv_file_list = glob.glob(