generate data for perform mask language model. :parameter source_file: source file where raw data come from :parameter target_file: save

positional wise fully connected feed forward implement as original version. FFN(x) = max(0,xW1+b1)W2+b2 this function provide

process one chunk for generate data of language model :return:

project the representation, then to do classification. :param h: [batch_size,sequence_length,d_model] :return: logits: [batch

project the representation, then to do classification. :param h: [batch_size,sequence_length,d_model] :return: logits: [batch

scaled dot product attention :param Q: query. shape:[batch,sequence_length,d_model] :param K_s: keys. shape:[batch,sequenc

based on the loss, use SGD to update parameter

based on the loss, use SGD to update parameter