* Fills token type and boundaries. */

Load tokenized code samples of one class i.e. source code files solving one programming problem The loaded samples a

Compute bag of tokens from list of tokens where <bag of tokens> is a vector of tokens frequences Parameters: - token

Make experimental two way convolutional Siamese dnn for sequence similarity analysis Parameters to each DNN are passed as se

Deal with DOS (\r \n) and classic Mac OS (\r) (physical) line endings. In case of CR LF skip (but count) the CR and return LF. In case of CR not

Deal with DOS (\r \n) and classic Mac OS (\r) line endings. Keeps track of physical coordinates and absolute location for each character. */

Deal with DOS (\r \n) and classic Mac OS (\r) (physical) line endings. In case of CR LF skip (but count) the CR and return LF. In case of CR not

Parse the JSON input (global) into an array of tokens. Returns the allocated token array. The number of tokens is passed back via num_tokens.

Print report on performed tokenization

emit NEWLINE and deal with indentation

Read datset problems and solutions from csv file Parameters; - cvs_file -- csv file defining dataset Returns: - dictionary defin

Write down report on statistics of generated dataset Parameters: - val_start -- index of start problem for validation

Report misclassifications of given classes Parameters: - classes -- indices of classes to report - mistakes -- list

Report errors in similarity detection of problem solutions Parameters: - sim_accuracy -- list of accuracy of similarity dete

Make two way convolutional Siamese dnn for sequence similarity analysis Parameters: - n_tokens -- number o

Convert similarity metric to probabilities Parameters: - sim_metric -- similarity metric - labels01 -- label t

Makes sure there is room in the token buffer.

Tokenize a given file Parameters: - filename -- name of source code file to tokenize - tokenizer -- path to tokenizer executable

Compute sizes of training and validation datasets rounded to batch size Use only samples left after reserving samples for te

Make training and validation tf.datasets by splitting loaded data set The tf.data.dataset is constructed from_generator

Write information on tokenization into the dataset

Write down distribution of datasets lables

Report problem solutions whose similarity was incorrectly classified Parameters: - mistakes -- list of indices o

Escape token for output as CSV string.

Escape token for output as CSV string.

Escape token for output as CSV string.

Escape token for output as CSV string.

Escape token for output as JSON string.

Escape token for output as JSON string.

Escape token for output as JSON string.

Escape hard newlines in a string.

Expand the given dataset with pairs of dissimilar solutions Parameters: - ds -- dataset for adding samples t

Add samples with similar solutions of the problems to the given dataset Parameters: - ds -- dataset f

Construct a convolutional layer Regularization is applied with self._regularizer Parameters: - input_tensor -- input

Experimental dense layer with fixed value of bias Parameters: - input_tensor -- input tensor - width -- widt

Experomental version of function for making functional dnn model and compiling it Parameters: - inputs -- list o

Make samples with categorical coding Parameters: - samples -- samples as list of sequences of tokens Returns:

Make lables from set of similarity samples Parameters: - annotations -- list of similarity samples in form of 4-tuples

Load sequence of tokens for solution source code Transforms string representation into int Parameters: - line -- desc

Add to the dnn a convolutional layer Parameters: - dnn -- dnn to buld - convolution -- specification of t

Add droput layer to dnn Parameters: - dnn -- dnn to buld - rate -- dropout rate If it is

Add pooling layer to the dnn Parameters: - dnn -- dnn to buld - pool -- pooling operation; either

Annotate samples with their problem indices and names

A Facebook Aroma type labeling scheme @param root @param rwdi

AromaLabeling stage one, just find out all the reserved words @param root @param rwdi

Input: data: PyG data object Output: data (edges are augmented in the following ways): data.e

Devide solutions of each problem evenly between test and, training and test datasets Returns: - test dataset

Devide solutions of each problem evenly between validation and training datasets Parameters: - valpart -- Fractio

index the tree via layer-wise traversal (i.e. BFS) @param root

Experimental dense layer with fixed value of bias Parameters: - input_tensor -- input tensor - width -- widt

Phase 3. Buffers up enough characters to encompass any possible token length. Assumes buffer contents is white-space followed by at least 1 toke