Class LogisticRegression

numpy_ml/linear_models/logistic.py:5–168

class LogisticRegression:
    def __init__(self, penalty="l2", gamma=0, fit_intercept=True):
        r"""
        A simple binary logistic regression model fit via gradient descent on
        the penalized negative log likelihood.

        Notes
        -----
        In simple binary logistic regression, the entries in a binary target
        vector :math:`\mathbf{y} = (y_1, \ldots, y_N)` are assumed to have been
        drawn from a series of independent Bernoulli random variables with
        expected values :math:`p_1, \ldots, p_N`. The binary logistic regession
        model models the logit of these unknown mean parameters as a linear
        function of the model coefficients, :math:`\mathbf{b}`, and the
        covariates for the corresponding example, :math:`\mathbf{x}_i`:

        .. math::

            \text{Logit}(p_i) =
                \log \left( \frac{p_i}{1 - p_i} \right) = \mathbf{b}^\top\mathbf{x}_i

        The model predictions :math:`\hat{\mathbf{y}}` are the expected values
        of the Bernoulli parameters for each example:

        .. math::

            \hat{y}_i =
                \mathbb{E}[y_i \mid \mathbf{x}_i] = \sigma(\mathbf{b}^\top \mathbf{x}_i)

        where :math:`\sigma` is the logistic sigmoid function :math:`\sigma(x)
        = \frac{1}{1 + e^{-x}}`. Under this model, the (penalized) negative log
        likelihood of the targets **y** is

        .. math::

            - \log \mathcal{L}(\mathbf{b}, \mathbf{y}) = -\frac{1}{N} \left[
                \left(
                    \sum_{i=0}^N y_i \log(\hat{y}_i) +
                      (1-y_i) \log(1-\hat{y}_i)
                \right) - R(\mathbf{b}, \gamma)
            \right]

        where

        .. math::

            R(\mathbf{b}, \gamma) = \left\{
                \begin{array}{lr}
                    \frac{\gamma}{2} ||\mathbf{b}||_2^2 & :\texttt{ penalty = 'l2'}\\
                    \gamma ||\mathbf{b}||_1 & :\texttt{ penalty = 'l1'}
                \end{array}
                \right.

        is a regularization penalty, :math:`\gamma` is a regularization weight,
        `N` is the number of examples in **y**, :math:`\hat{y}_i` is the model
        prediction on example *i*, and **b** is the vector of model
        coefficients.