add a new column to the synthetic data generation specification :param colName: Name of column to add. If this conflicts with the underlying

build the test data set from the column definitions and return a dataframe for it if `withStreaming` is True, generates a streaming data set

output seed column field (defaults to `id`) as a column in the generated data set if specified If this is not called, the seed column field

add a column specification for an existing column :returns: modified in-place instance of test data generator allowing for chaining of calls

Get a table generator from the dataset provider These are DataGenerator instances that can be used to generate the data. The dataset

get column spec for column having name supplied :param name: name of column to find spec for :return: column spec for named column i

populate column definitions and specifications for each of the columns in the schema :param sch: Spark SQL schema, from which fields are add

Create a machine local Spark instance for Datalib. By default, it uses `n-1` cores of the available cores for the spark session, wher

ensure(cond, s) => throws Exception(s) if c is not true :param cond: condition to test :param msg: Message to add to exception if exception i

check if field1 is built before field2

Make a clone of the data spec via deep copy preserving same spark session :returns: deep copy of test data generator definition

check if field1 is built in separate phase to field2

get the `max` attribute

get list of output columns by flattening list of lists of column names normal columns will have a single column name but column definitio

get the `min` attribute

Explain the test data generation process :param suppressOutput: If True, suppress display of build plan :returns: String containing e

Add a struct column to the synthetic data generation specification. This will add a new column composed of a struct of the specified

Add a constraint to control the data generation :param constraint: a constraint object to apply to the data generation :returns: refe

Set the random seed for the text generator :param seed: seed value to set :return: self

get the `expr` attributed used to generate values for this column

Get the dataset definition for the class

set option to option value for later processing :param optionKey: key for option :param optionValue: value for option :retur

Get value from data summary :param dataSummary: Data summary to search, optional :param colName: Column name of column to get value

Add a measure to the summary dataframe :param measureName: Name of measure :param summaryExpr: Summary expression :param fie

Compute the number of partitions based on rows and columns :param rows: number of rows :param columns: number of columns :re

For a supplied list of arguments, returns the first argument that does not have the value `None` :param args: variable list of arguments which ar

Get the registered dataset definitions :return: A dictionary of registered datasets metadata objects

Add a constraint to control the data generation :param constraints: a list of constraint objects to apply to the data generation :ret

prepare for building by computing a pseudo build plan The build plan is not a true build plan - it is only used for debugging purposes, but

Convert range to discrete range :returns: number of discrete values in range. For example `NRange(1, 5, 0.5)` has 8 discrete values

Generate a SQL expression that combines multiple constraints using AND :param constraintExpressions: list of Pyspark SQL Column constraint e

Modify the row count - useful when starting a new spec from a clone :param rc: The count of rows to generate :returns: modified in-pl

Get columns as list of columns from string of list-like :param columns: string or list of strings representing column names

Get the dataset tables list for the class

Get numpy random number generator :return: returns random number generator initialized from previously supplied random seed

Get the keys as list of strings

Register the dataset provider type using metadata defined in the dataset provider :param datasetProvider: Dataset provider class :re

Add column specs for columns matching a) list of field names, b) one or more regex patterns c) type (as in pyspark.sq

Prepare the random bounds for processing of the template expansion For each template, we will have a vector of random numbers to gen

Generate a Spark SQL data type from a string Allowable options for `type_string` parameter are: * `string`, `varchar`, `char`, `nva

This method lists the registered datasets It filters the list by a regular expression pattern if provided :param name: name o

get list of output columns

Get val for key if it exists or else return default

Get numpy random number generator :param random_seed: Numeric random seed to use. If < 0, then no random :return:

entry point to use for pandas udfs Implementation uses vectorized implementation of process :param v: Pandas series of values passe

Python equivalent of Scala stripMargins method Takes a string (potentially multiline) and strips all chars up and including the first oc

Specifies context initialization function :param fn: function pointer or lambda function for initialization signa

get column names as list of strings

Get the registered datasets version indicator :return: A dictionary of registered datasets

Parse a schema from a schema string :param sparkSession: spark session to use :param source_schema: should be a table defini

generate create table script suitable for format of test data set :param name: name of table to use in generated script :param locat

If called, specifies the property of the context to be passed to the text generation function. If not called, the context object itself w

write underlined text in RST markup format :param outputFile: output file to write to :param text: text to write :param underline: char

check spark version :param sparkVersion: spark version string :param minSparkVersion: min spark version as tuple :ret

Get the default parallelism for a spark session, if spark session supports getting the sparkContext :param sparkSession: spark session

Determine min and max number of columns to generate along with `structType` for columns with multiple columns / features :param

generate random integer between low and high inclusive :param low: low value, if no high value is specified, treat low value as high value a

Get seed expression for column generation This is used to generate the base value for every column if using a single base column, th

Mark that build plan needs to be regenerated :returns: modified in-place instance of test data generator allowing for chaining of calls follo

flatten list :param lst: list to flatten

Check if object is a valid data provider type :param candidateDataProvider: potential Dataset provider class :return: True if valid D

process directory for package or subpackage :param outputFile: output file instance :param pathToProcess: path to process :param subpack

Check against Python version Allows minimum version to be passed in to facilitate unit testing :param python_version_expected: = minim

Generate summary analysis of data set as dataframe :return: Summary results as dataframe The resulting dataframe can be displayed w

convert string to Python DateTime object using format

Set up logging This will set the logger at warning, info or debug levels depending on the instance construction parameters

Create enter / exit object to support temporary renaming of column spec This is to support the functionality: ``` with c

Check that option is either not specified or of type boolean :param v: value to test :param name: name of value to use in any report

determine smallest numpy type to represent values :param listValues: list or np.ndarray of values to get np.dtype for :return: np.dt

Formats supplied code as Html suitable for use with notebook ``displayHTML`` :param codeText: Code to be wrapped in html section :re

get a name for the data set Uses the untitled name prefix and nextNameIndex to generate a dummy dataset name :returns: stri

generate text for seed based on configuration parameters. As it uses numpy, repeatability is restricted depending on version of the

get value v as tuple or return default value :param v: value to test :param defaultValue: value to use as a default if value

Get column Spark SQL datatype for specified column :param colName: name of column as string :returns: Spark SQL datatype for named c

Get provider definitions for one or more datasets :param name: name of dataset to get provider for, if None, returns all providers :p

Get scale of range

gets count of errors containing specified text :param caplog_object: log capture object from fixture :param searchText: text

Check is all instance vars are populated :returns: `True` if fully populated, `False` otherwise

parse time interval from string

process list of items :param outputFile: output file instance :param items: list of items. each item is a tuple of ( "moduleName.typename",

generate merge table script suitable for format of test data set :param tgtName: name of target table to use in generated script :pa

set up log capture fixture Sets up log capture fixture to only capture messages after setup and only capture warnings and errors

Generate summary analysis of data set and return / print summary results :param suppressOutput: If False, prints results to console also

return system time as milliseconds since start of epoch :return: system time millis as long

Adjust for minimum value of data range :param baseval: base expression :param datarange: data range to conform to :param forc