Class Features

src/datasets/features/features.py:1847–2388

A special dictionary that defines the internal structure of a dataset. Instantiated with a dictionary of type `dict[str, FieldType]`, where keys are the desired column names, and values are the type of that column. `FieldType` can be one of the following: - [`Value`] feature sp

class Features(dict):
    """A special dictionary that defines the internal structure of a dataset.

    Instantiated with a dictionary of type `dict[str, FieldType]`, where keys are the desired column names,
    and values are the type of that column.

    `FieldType` can be one of the following:
        - [`Value`] feature specifies a single data type value, e.g. `int64` or `string`.
        - [`ClassLabel`] feature specifies a predefined set of classes which can have labels associated to them and
          will be stored as integers in the dataset.
        - Python `dict` specifies a composite feature containing a mapping of sub-fields to sub-features.
          It's possible to have nested fields of nested fields in an arbitrary manner.
        - [`List`] or [`LargeList`] specifies a composite feature containing a sequence of
          sub-features, all of the same feature type.
        - [`Array2D`], [`Array3D`], [`Array4D`] or [`Array5D`] feature for multidimensional arrays.
        - [`Audio`] feature to store the absolute path to an audio file or a dictionary with the relative path
          to an audio file ("path" key) and its bytes content ("bytes" key).
          This feature loads the audio lazily with a decoder.
        - [`Image`] feature to store the absolute path to an image file, an `np.ndarray` object, a `PIL.Image.Image` object
          or a dictionary with the relative path to an image file ("path" key) and its bytes content ("bytes" key).
          This feature extracts the image data.
        - [`Video`] feature to store the absolute path to a video file, a `torchcodec.decoders.VideoDecoder` object
          or a dictionary with the relative path to a video file ("path" key) and its bytes content ("bytes" key).
          This feature loads the video lazily with a decoder.
        - [`Pdf`] feature to store the absolute path to a PDF file, a `pdfplumber.pdf.PDF` object
          or a dictionary with the relative path to a PDF file ("path" key) and its bytes content ("bytes" key).
          This feature loads the PDF lazily with a PDF reader.
        - [`Nifti`] feature to store the absolute path to a NIfTI neuroimaging file, a `nibabel.Nifti1Image` object
          or a dictionary with the relative path to a NIfTI file ("path" key) and its bytes content ("bytes" key).
          This feature loads the NIfTI file lazily with nibabel.
        - [`Translation`] or [`TranslationVariableLanguages`] feature specific to Machine Translation.
        - [`Json`] feature to store unstructred data, e.g. containing mixed/abritrary types. Under the hood
    """

    def __init__(*args, **kwargs):
        # self not in the signature to allow passing self as a kwarg
        if not args:
            raise TypeError("descriptor '__init__' of 'Features' object needs an argument")
        self, *args = args
        super(Features, self).__init__(*args, **kwargs)
        # keep track of columns which require decoding
        self._column_requires_decoding: dict[str, bool] = {
            col: require_decoding(feature) for col, feature in self.items()
        }

        # backward compatibility with datasets<4 : [feature] -> List(feature)
        def _check_old_list(feature):
            if isinstance(feature, list):
                return List(_visit(feature[0], _check_old_list))
            return feature

        for column_name, feature in self.items():
            self[column_name] = _visit(feature, _check_old_list)

    __setitem__ = keep_features_dicts_synced(dict.__setitem__)
    __delitem__ = keep_features_dicts_synced(dict.__delitem__)
    update = keep_features_dicts_synced(dict.update)
    setdefault = keep_features_dicts_synced(dict.setdefault)