Function parse_keystats

parsing_keystats.py:88–222

We have downloaded a large number of html files, which are snapshots of a ticker at different times, containing the fundamental data (our features). To extract the key statistics, we use regex. For supervised machine learning, we also need the data that will form our dependent variable,

(sp500_df, stock_df)

def parse_keystats(sp500_df, stock_df):
    """
    We have downloaded a large number of html files, which are snapshots of a ticker at different times,
    containing the fundamental data (our features). To extract the key statistics, we use regex.
    For supervised machine learning, we also need the data that will form our dependent variable,
    the performance of the stock compared to the SP500.
    :sp500_df: dataframe containing SP500 prices
    :stock_df: dataframe containing stock prices
    :return: a dataframe of training data (i.e features and the components of our dependent variable)
    """
    # The tickers whose data is to be parsed.
    stock_list = [x[0] for x in os.walk(statspath)]
    stock_list = stock_list[1:]

    # Creating a new dataframe which we will later fill.
    df_columns = [
        "Date",
        "Unix",
        "Ticker",
        "Price",
        "stock_p_change",
        "SP500",
        "SP500_p_change",
    ] + features

    df = pd.DataFrame(columns=df_columns)

    # tqdm is a simple progress bar
    for stock_directory in tqdm(stock_list, desc="Parsing progress:", unit="tickers"):
        keystats_html_files = os.listdir(stock_directory)

        # Snippet to get rid of the .DS_Store file in macOS
        if ".DS_Store" in keystats_html_files:
            keystats_html_files.remove(".DS_Store")

        ticker = stock_directory.split(statspath)[1]

        for file in keystats_html_files:
            # Convert the datetime format of our file to unix time
            date_stamp = datetime.strptime(file, "%Y%m%d%H%M%S.html")
            unix_time = time.mktime(date_stamp.timetuple())

            # Read in the html file as a string.
            full_file_path = stock_directory + "/" + file

            # This will store the parsed values
            value_list = []

            with open(full_file_path, "r") as source:
                source = source.read()
                # Remove commas from the html to make parsing easier.
                source = source.replace(",", "")

                # Regex search for the different variables in the html file, then append to value_list
                for variable in features:
                    # Search for the table entry adjacent to the variable name.
                    try:
                        regex = (