BiLSTM-CNN-CRF tagger is a PyTorch implementation of "mainstream" neural tagging scheme based on works of Lample, et. al., 2016 and Ma et. al., 2016.
|__ articles/ --> collection of papers related to the tagging, argument mining, etc.
|__ classes/
|__ data_io.py --> class for reading/writing data in different CoNNL file formats
|__ datasets_bank.py --> class for storing the train/dev/test data subsets and sampling batches
from the train dataset
dataset
|__ evaluator.py --> class for evaluation of F1 scores and token-level accuracies
|__ report.py --> class for storing the evaluation results as text files
|__ tag_components.py --> class for extracting tag components from BOI encodings
|__ utils.py --> several auxiliary utils and functions
|__ data/
|__ NER/ --> Datasets for Named Entity Recognition
|__ CoNNL_2003_shared_task/ --> data for NER CoNLL-2003 shared task (English) in BOI-2
CoNNL format, from E.F. Tjong Kim Sang and F. De Meulder,
Introduction to the CoNLL-2003 Shared Task:
Language-Independent Named Entity Recognition, 2003.
|__ AM/ --> Datasets for Argument Mining
|__ persuasive_essays/ --> data for persuasive essays in BOI-2-like CoNNL format, from:
Steffen Eger, Johannes Daxenberger, Iryna Gurevych. Neural
End-to-End Learning for Computational Argumentation Mining, 2017
|__ embeddings/
|__ get_glove_embeddings.sh --> script for downloading GloVe6B 100-dimensional word embeddings
|__ layers/
|__ layer_base.py --> abstract base class for all types of layers
|__ layer_birnn_base.py --> abstract base class for all bidirectional recurrent layers
|__ layer_word_embeddings.py --> class implements word embeddings
|__ layer_char_embeddings.py --> class implements character-level embeddings
|__ layer_char_cnn.py --> class implements character-level convolutional 1D operation
|__ layer_bilstm.py --> class implements bidirectional LSTM recurrent layer
|__ layer_bigru.py --> class implements bidirectional GRU recurrent layer
|__ layer_crf.py --> class implements conditional random field (CRF)
|__ models/
|__ tagger_base.py --> abstract base class for all types of taggers
|__ tagger_io.py --> contains wrappers to create and load tagger models
|__ tagger_birnn.py --> vanilla BiLSTM/BiGRU tagger model
|__ tagger_birnn_crf.py --> BiLSTM/BiGRU + CRF tagger model
|__ tagger_birnn_cnn.py --> BiLSTM/BiGRU + char-level CNN tagger model
|__ tagger_birnn_cnn_crf.py --> BiLSTM/BiGRU + char-level CNN + CRF tagger model
|__ pretrained/
|__ tagger_NER.hdf5 --> tagger for NER, BiGRU+CNN+CRF trained on NER-2003 shared task, English
|__ seq_indexers/
|__ seq_indexer_base.py --> abstract class for sequence indexers, they converts list of lists
of string items
to the list of lists of integer indices and back
|__ seq_indexer_base_embeddings.py --> abstract sequence indexer class that implements work
with embeddings
|__ seq_indexer_word.py --> converts list of lists of words as strings to list of lists of
integer indices and back, has built-in embeddings
|__ seq_indexer_char.py --> converts list of lists of characters to list of lists of integer
indices and back, has built-in embeddings
|__ seq_indexer_tag.py --> converts list of lists of string tags to list of lists of integer
indices and back, doesn't have built-in embeddings
|__ main.py --> main script for training/evaluation/saving tagger models
|__ run_tagger.py --> run the trained tagger model from the checkpoint file
|__ conlleval --> "official" Perl script from NER 2003 shared task for evaluating the f1 scores,
author: Erik Tjong Kim Sang, version: 2004-01-26
|__ requirements.txt --> file for managing packages requirements
Results of training the models with the default settings:
| tagger model | dataset | micro-f1 on test |
|---|---|---|
| BiLSTM + CNN + CRF Lample et. al., 2016 | NER-2003 shared task (English) | 90.94 |
| BiLSTM + CNN + CRF Ma et al., 2016 | NER-2003 shared task (English) | 91.21 |
| BiLSTM + CNN + CRF (our) | NER-2003 shared task (English) | 90.86 |
| STag_BLCC, Eger et. al., 2017 | AM Persuasive Essays, Paragraph Level | 66.69 |
| LSTM-ER, Eger et. al., 2017 | AM Persuasive Essays, Paragraph Level | 70.83 |
| BiGRU + CNN + CRF (our) | AM Persuasive Essays, Paragraph Level | 64.31 |
In order to ensure the consistency of the experiments, for evaluation purposes we use "official" Perl script from NER 2003 shared task, author: Erik Tjong Kim Sang, version: 2004-01-26, example of it's output:
processed 46435 tokens with 5648 phrases; found: 5679 phrases; correct: 5146.
accuracy: 97.92%; precision: 90.61%; recall: 91.11%; FB1: 90.86
LOC: precision: 91.35%; recall: 93.65%; FB1: 92.48 1710
MISC: precision: 78.20%; recall: 82.76%; FB1: 80.42 743
ORG: precision: 90.25%; recall: 88.02%; FB1: 89.12 1620
PER: precision: 95.95%; recall: 95.30%; FB1: 95.63 1606
To train/evaluate/save trained tagger model, please run the main.py script.
usage: main.py [-h] [--seed_num SEED_NUM] [--model MODEL]
[--fn_train FN_TRAIN] [--fn_dev FN_DEV] [--fn_test FN_TEST]
[--load LOAD] [--save SAVE] [--wsi WSI] [--emb_fn EMB_FN]
[--emb_dim EMB_DIM] [--emb_delimiter EMB_DELIMITER]
[--freeze_word_embeddings FREEZE_WORD_EMBEDDINGS]
[--freeze_char_embeddings FREEZE_CHAR_EMBEDDINGS] [--gpu GPU]
[--check_for_lowercase CHECK_FOR_LOWERCASE]
[--epoch_num EPOCH_NUM] [--min_epoch_num MIN_EPOCH_NUM]
[--patience PATIENCE] [--rnn_type RNN_TYPE]
[--rnn_hidden_dim RNN_HIDDEN_DIM]
[--char_embeddings_dim CHAR_EMBEDDINGS_DIM]
[--word_len WORD_LEN]
[--char_cnn_filter_num CHAR_CNN_FILTER_NUM]
[--char_window_size CHAR_WINDOW_SIZE]
[--dropout_ratio DROPOUT_RATIO] [--dataset_sort DATASET_SORT]
[--clip_grad CLIP_GRAD] [--opt_method OPT_METHOD]
[--batch_size BATCH_SIZE] [--lr LR] [--lr_decay LR_DECAY]
[--momentum MOMENTUM] [--verbose VERBOSE]
[--match_alpha_ratio MATCH_ALPHA_RATIO] [--save_best SAVE_BEST]
[--report_fn REPORT_FN]
Learning tagging problem using neural networks
optional arguments:
-h, --help show this help message and exit
--seed_num SEED_NUM Random seed number, you may use any but 42 is the
answer.
--model MODEL Tagger model: "BiRNN", "BiRNNCNN", "BiRNNCRF",
"BiRNNCNNCRF".
--fn_train FN_TRAIN Train data in CoNNL-2003 format.
--fn_dev FN_DEV Dev data in CoNNL-2003 format, it is used to find best
model during the training.
--fn_test FN_TEST Test data in CoNNL-2003 format, it is used to obtain
the final accuracy/F1 score.
--load LOAD Path to load from the trained model.
--save SAVE Path to save the trained model.
--wsi WSI Load word_seq_indexer object from hdf5 file.
--emb_fn EMB_FN Path to word embeddings file.
--emb_dim EMB_DIM Dimension of word embeddings file.
--emb_delimiter EMB_DELIMITER
Delimiter for word embeddings file.
--freeze_word_embeddings FREEZE_WORD_EMBEDDINGS
False to continue training the \ word embeddings.
--freeze_char_embeddings FREEZE_CHAR_EMBEDDINGS
False to continue training the char embeddings.
--gpu GPU GPU device number, 0 by default, -1 means CPU.
--check_for_lowercase CHECK_FOR_LOWERCASE
Read characters caseless.
--epoch_num EPOCH_NUM
Number of epochs.
--min_epoch_num MIN_EPOCH_NUM
Minimum number of epochs.
--patience PATIENCE Patience for early stopping.
--rnn_type RNN_TYPE RNN cell units type: "Vanilla", "LSTM", "GRU".
--rnn_hidden_dim RNN_HIDDEN_DIM
Number hidden units in the recurrent layer.
--char_embeddings_dim CHAR_EMBEDDINGS_DIM
Char embeddings dim, only for char CNNs.
--word_len WORD_LEN Max length of words in characters for char CNNs.
--char_cnn_filter_num CHAR_CNN_FILTER_NUM
Number of filters in Char CNN.
--char_window_size CHAR_WINDOW_SIZE
Convolution1D size.
--dropout_ratio DROPOUT_RATIO
Dropout ratio.
--dataset_sort DATASET_SORT
Sort sequences by length for training.
--clip_grad CLIP_GRAD
Clipping gradients maximum L2 norm.
--opt_method OPT_METHOD
Optimization method: "sgd", "adam".
--batch_size BATCH_SIZE
Batch size, samples.
--lr LR Learning rate.
--lr_decay LR_DECAY Learning decay rate.
--momentum MOMENTUM Learning momentum rate.
--verbose VERBOSE Show additional information.
--match_alpha_ratio MATCH_ALPHA_RATIO
Alpha ratio from non-strict matching, options: 0.999
or 0.5
--save_best SAVE_BEST
Save best on dev model as a final model.
--report_fn REPORT_FN
Report filename.
usage: run_tagger.py [-h] [--fn FN] [--checkpoint_fn CHECKPOINT_FN]
[--gpu GPU]
Run trained tagger from the checkpoint file
optional arguments:
-h, --help show this help message and exit
--fn FN Train data in CoNNL-2003 format.
--checkpoint_fn CHECKPOINT_FN
Path to load the trained model.
--gpu GPU GPU device number, 0 by default, -1 means CPU.
``` Evaluation
batch_size=10 char_cnn_filter_num=30 char_embeddings_dim=25 char_window_size=3 check_for_lowercase=True clip_grad=5 dataset_sort=True dropout_ratio=0.5 emb_delimiter=' ' emb_dim=100 emb_fn='embeddings/glove.6B.100d.txt' epoch_num=100 fn_dev='data/NER/CoNNL_2003_shared_task/dev.txt' fn_test='data/NER/CoNNL_2003_shared_task/test.txt' fn_train='data/NER/CoNNL_2003_shared_task/train.txt' freeze_char_embeddings=False freeze_word_embeddings=False gpu=0 load=None lr=0.01 lr_decay=0.05 match_alpha_ratio=0.999 min_epoch_num=50 model='BiRNNCNNCRF' momentum=0.9 opt_method='sgd' patience=20 report_fn='2018_10_09_07-55_14_report.txt' rnn_hidden_dim=100 rnn_type='LSTM' save='2018_10_09_07-55_14_tagger.hdf5' save_best=False seed_num=42 verbose=True word_len=20 wsi=None
epoch | train loss | f1-train | f1-dev | f1-test | acc. train | acc. dev | acc. test
1 | 302.08 | 82.69 | 83.02 | 80.19 | 95.68 | 95.59 | 95.20
2 | 151.72 | 89.32 | 88.63 | 84.90 | 97.66 | 97.43 | 96.56
3 | 108.10 | 91.76 | 90.80 | 87.84 | 98.35 | 98.08 | 97.37
4 | 88.41 | 92.41 | 90.64 | 88.01 | 98.51 | 98.11 | 97.44
5 | 75.45 | 93.66 | 91.76 | 89.20 | 98.76 | 98.28 | 97.53
6 | 67.20 | 94.45 | 92.35 | 89.94 | 98.92 | 98.40 | 97.78
7 | 61.48 | 95.35 | 92.96 | 89.94 | 99.10 | 98.53 | 97.78
8 | 56.26 | 95.38 | 92.34 | 89.62 | 99.11 | 98.44 | 97.67
9 | 52.61 | 95.68 | 92.35 | 89.43 | 99.16 | 98.44 | 97.57
10 | 48.84 | 96.44 | 93.18 | 90.20 | 99.31 | 98.61 | 97.77
11 | 45.93 | 96.53 | 92.79 | 90.10 | 99.34 | 98.52 | 97.76
12 | 42.84 | 96.71 | 93.12 | 89.99 | 99.33 | 98.53 | 97.59
13 | 40.87 | 97.11 | 93.34 | 90.31 | 99.46 | 98.64 | 97.79
14 | 39.28 | 97.32 | 93.51 | 90.39 | 99.49 | 98.66 | 97.81
15 | 37.28 | 97.51 | 93.50 | 90.32 | 99.53 | 98.67 | 97.84
16 | 35.54 | 9
$ claude mcp add targer \
-- python -m otcore.mcp_server <graph>