Computes the probability Pr(y | votes) using Bayes Rule over Pr(votes, y). Args: - votes: m element array of votes in {-1, 0, ..., k

Naive bayes estimation. Estimate Pr(vote_i | y) from training data and use that to compute Pr(y = 1 | votes). Assumes conditional in

Take a majority vote over predictors. Current implementation ignores abstains. When there is a tie, we pick the prompt with the lowest index.

Computes the probability Pr(y = 1 | votes).

Use the predictor with the best performance on the train set.

Creates a data structure c_data which maps cliques and separator sets to their maximal clique.

Constructs a data structure c_tree that contains nodes and edges of the junction tree. Args: edgeset: List of tuples (i,

FlyingSquid algorithm requires no labeled data (except for estimating class balance). Assumes conditional independence (for now) and symmetr

Computes the probability Pr(votes | y).

Computes marginal probability over voters indexed by idx, Pr(votes_idxs = vals | y).

Junction tree estimation. Estimate Pr(vote_i | y) from training data and use that to compute Pr(y = 1 | votes). Assumes edgeset stru

Data programming algorithm. Args: - with_label: Do we use y or not? If using label, use the train set and do MLE on Pr(y, votes);

Computes the probability Pr(votes, y) assuming conditional independence. Args: - votes: m element array of votes in {-1, 0, ..., k-1

Using our learned PGM, output rounded estimates of Pr(y = 1 | votes) and computes its accuracy. Args: test_v

Computes marginal probability over votes indexed by idx, Pr(votes_idxs = vals | y), using training data.

Computes the probability Pr(votes | y).

To counteract the large model hallucinating on various chunks affecting the evaluation of good functions.

Compute average F1 of text spans. Taken from Squad without prob threshold for no answer.

Compute average F1 of text spans. Taken from Squad without prob threshold for no answer.

Computes Pr(vote_i | y) on training data. Each prompt has k x k values. We ignore the abstaining case Pr(vote_i = 0 | y), since this is handl

Estimates the class balance Pr(y) on training data. Sets self.balance to be an array of length k.

Computes Pr(vote_i != 0) (coverage) and Pr(vote_i = 0 | y) for each y (abstain_rate).

Estimates Pr(vote_i | y = 0, 1) using FlyingSquid algorithm. Args: - on_test: If we use the unlabeled test dataset or the labeled tr

Computes Pr(vote_i | y) on training data similarly to above, but assumes Pr(vote_i = c | y = c) = Pr(vote_i = y), independent of what the val

self.accs[k, i, j] = Pr(lf_k = j | y = i) (i, j scaled to -1, 1 if needed)

self.accs[k, i, j] = Pr(lf_k = j | y = i) (i, j scaled to -1, 1 if needed)

For binary tasks defined with classes [0, 1] and abstain -1, we construct scaled versions with classes [-1, 1] and abstain 0. Scaled

Compute Pr(vote_i = y) on the test data.

Compute Pr(vote_i = y) on the training data.

Computes -1/n sum_i log Pr(y-hat = y, votes_j), minimizing cross entropy over the joint distribution of Y, votes.

Computes -1/n sum_i log Pr(y-hat = y | votes_j). This is the standard notion of CE loss.

Computes -1/n sum_j log Pr(votes_j), minimizing cross entropy over the distribution of votes