Editor for Treebanks in CoNLL-U format and Front-End for dependency parser servers

This Software is a tool which facilitates the editing of syntactic relations and morphological features of files in CoNLL-U format (http://universaldependencies.org/format.html). It uses a Java-based server and a HTML/CSS/Javascript based front-end. The editor loads the CoNLL-U file and saves changes to disk (and performs a git commit if the file is under git version control).

The editor provides the following functionalities: * editing words (forms, lemmas, upos, xpos, features, enhanced dependencies) (fast edit for UPOS and deprel) * editing dependency relations * indicates invalid UPOS, XPOS or dependency relations * join/split/delete words (to correct tokenization errors) * join/split sentences (to correct segmentation errors) * undo/redo (partially) * search: forms, lemmas, UPOS, XPOS, deprels, sentences IDs and comments, sequences of any of these, searching for subtrees, importing subtrees from current sentence, sd-parse support * searching by source file line numbers * edit non-CoNLL-U columns in a subset of CoNLL-U plus files * create multiword tokens from existing words or add a MWT to contract two ore more existing words * git support * export of dependency graphs as svg or LaTeX (for the tikz-dependency package or the doc/deptree.sty class, see documentation) * prohibits invalid (cyclic) trees * Three edit modes: dependency trees, dependency «hedges» and a table edit mode * mass editing: modify tokens if a (complex) condition is satisfied * validation (using implications: if conditions1 true then conditions2 must be true) * sentence metadata editing * add highlight information on tokens and deprels which have yet to be validated (see highlighting) * adding Translit= values to the MISC column (transliterating the FORM column) see section Transliteration * finding similar or identical sentence in a list of CoNLL-U files, see section Find Similar Sentences * configuring the UI on order to hide unneeded functionalities which otherwise clutter the UI

Current version: 2.33.1 (see change history)

ConlluEditor can also be used as front-end to display the results of dependency parsing in the same way as the editor. * dependency tree/dependency hedge * CoNLL-U/LaTeX/SD-Parse format

For more information see section Parser Front-End

In order to compare two files (e.g. a gold file with a predicted file) ConlluEditor provides * a file compare mode

For more information see section File Comparison

Installation

Requirements

• Java jre >= 17.0
• Firefox (tested with version 60.9 on Windows 10, >= 63 on Linux and 69.0.1 on MacOS Sierra),
• Chromium or Chrome (both tested with version 70 on Linux),
• Edge (tested with version 44.17763.1.0 on Windows 10),
• Opera (tested with version 63 on Linux),
• Safari (tested with version 11.1 on MacOS Sierra)
• jquery 3.3.1 (https://code.jquery.com/jquery-3.3.1.min.js) and jquery-ui 1.14.1 (https://jqueryui.com)
• bootstrap 4.1.3 (https://github.com/twbs/bootstrap/releases/download/v4.1.3/bootstrap-4.1.3-dist.zip)
• popper.min.js and popper.min.js.map 1.14.6 (https://unpkg.com/popper.js/dist/umd/popper.min.js{.map}), needed by bootstrap
• sorttable.js (https://www.kryogenix.org/code/browser/sorttable/sorttable.js)
• on MacOS install in addition also: greadlink (brew install coreutils)

In order two compile the server, you also need * Java jdk >= 17.0 * maven (tested with >= 3.3.9)

Docker

Alternatively, a recent version of Docker can be used to run the docker image (see section docker below)

Github Release

To avoid compilation, you can download the latest release, unzip the conllueditor-a.b.c.zip into a directory and start the server as described below.

License

• This software is under the 3-Clause BSD License

Compilation

• mvn install
• if the tests fail after git pull try deleting target/test-classes and testoutput. If this does not solve the problem, run mvn install -DskipTests to avoid running the JUnit tests and drop me a note
• individual tests can be run by mvn test -Dtest=TestConlluEditor#<testmethod>)
• redirect stderr in order to see only maven messages (test reports are generated in target/surefire-reports/):
  • mvn install 2> stderr_output.txt
  • mvn test -Dtest=TestConlluEditor#<testmethod> 2> stderr_output.txt
• download javascript libraries (bootstrap, popper, jquery and jquery-ui), the script bin/installJQ.sh will do the job
• or change the corresponding lines in gui/index.html to load the libraries from a public server:

<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js">

• after having run bin/installJQ.sh the gui/lib directory must look like this

./gui/lib/bootstrap-4.1.3/js/bootstrap.min.js
./gui/lib/bootstrap-4.1.3/js/bootstrap.bundle.js.map
./gui/lib/bootstrap-4.1.3/js/bootstrap.js.map
./gui/lib/bootstrap-4.1.3/js/bootstrap.bundle.min.js.map
./gui/lib/bootstrap-4.1.3/js/bootstrap.bundle.js
./gui/lib/bootstrap-4.1.3/js/bootstrap.js
./gui/lib/bootstrap-4.1.3/js/bootstrap.bundle.min.js
./gui/lib/bootstrap-4.1.3/js/bootstrap.min.js.map
./gui/lib/bootstrap-4.1.3/css/bootstrap-grid.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap.min.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap-grid.min.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap-reboot.min.css
./gui/lib/bootstrap-4.1.3/css/bootstrap-reboot.css.map
./gui/lib/bootstrap-4.1.3/css/bootstrap-grid.min.css
./gui/lib/bootstrap-4.1.3/css/bootstrap-grid.css
./gui/lib/bootstrap-4.1.3/css/bootstrap.min.css
./gui/lib/bootstrap-4.1.3/css/bootstrap-reboot.css
./gui/lib/bootstrap-4.1.3/css/bootstrap.css
./gui/lib/bootstrap-4.1.3/css/bootstrap-reboot.min.css.map
./gui/lib/jquery-3.3.1.min.js
./gui/lib/jquery-ui-1.14.1/jquery-ui.structure.min.css
./gui/lib/jquery-ui-1.14.1/external
./gui/lib/jquery-ui-1.14.1/external/jquery
./gui/lib/jquery-ui-1.14.1/external/jquery/jquery.js
./gui/lib/jquery-ui-1.14.1/jquery-ui.structure.css
./gui/lib/jquery-ui-1.14.1/package.json
./gui/lib/jquery-ui-1.14.1/jquery-ui.min.css
./gui/lib/jquery-ui-1.14.1/jquery-ui.js
./gui/lib/jquery-ui-1.14.1/jquery-ui.theme.min.css
./gui/lib/jquery-ui-1.14.1/jquery-ui.css
./gui/lib/jquery-ui-1.14.1/AUTHORS.txt
./gui/lib/jquery-ui-1.14.1/jquery-ui.theme.css
./gui/lib/jquery-ui-1.14.1/jquery-ui.min.js
./gui/lib/jquery-ui-1.14.1/index.html
./gui/lib/jquery-ui-1.14.1/images
./gui/lib/jquery-ui-1.14.1/images/ui-icons_cc0000_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_ffffff_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_777777_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_777620_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_444444_256x240.png
./gui/lib/jquery-ui-1.14.1/images/ui-icons_555555_256x240.png
./gui/lib/jquery-ui-1.14.1/LICENSE.txt
./gui/lib/popper.min.js
./gui/lib/popper.min.js.map

The server cannot be started without these libraries, an error message will show which files are missing.

Update ConlluEditor

to get bug corrections/new features, run

git pull
rm -rf target/test-classes
mvn install

Starting the server

On smaller machines, the memory management of the java VM (-Xmx... option) may have to be modified in bin/conlluedit.sh. The current value (-Xmx4g) is largely sufficient to load larger treebanks with up to 1,5M tokens.

ConlluEditor comes with a simple HTTP server:

bin/conlluedit.sh treebank.conllu 8888

Point your navigator to http://localhost:8888 .

Using docker

If you prefer a docker image, you can use the following

• get current docker image

docker pull jheinecke/conllueditor:latest

Run the image in a docker container from the directory where your .conllu-file (and other files like lists of UPOS, XPOS etc) reside and replace * --user 1000:1000 with your uid and gid (you get the uid on Linux with the id-command) * --env filename=<yourfile>.conllu with the filename you want to edit * </absolute/path/to/datadir> with the directory where the .conllu-file and other files reside

docker run --rm -t -d \
  --name conllueditor -p 8888:5555 \
    --user 1000:1000 \
    -v </absolute/path/to/datadir>:/data \
    --env filename=<yourfile>.conllu \
    jheinecke/conllueditor:latest

Note that the option -d causes the docker container to run in the background and error messages will not appear on screen unless you run docker logs conllueditor. If you want to see any messages the server outputs to the terminal, omit the -d.

Other parameters (shown below in section other options) can be given with --env <optionname>=value e.g. --env UPOS=uposfile.txt, --env UPOS=cpos.ud, --env XPOS=xpos.txt, --env language=cy, --env include_unused=1, --env deprels=deprels.ud, --env features=feat_val.ud, --env shortcuts=multi-shortcuts.json, --env compare=file.conllu, or --env uiconfig=uiconfig.yml . However all files given, must reside in the </absolute/path/to/datadir> directory. Further files given in the uiconfig.yml must also reside in the </absolute/path/to/datadir> directory.

When finished, stop the docker container (it is removed automatically by the --rm option given):

docker stop conllueditor

Other options

• --uiconfig <file> yaml file to specify activated buttosn, display etc (see below in section UI configuration)
• --UPOS <file> comma separated list of files containing valid UPOS tags (see https://github.com/UniversalDependencies/tools/tree/master/data/upos.json)
• --XPOS <file> comma separated list of files containing valid XPOS tags
• --deprels <file> comma separated list of files, containing valid dependency relation names (see https://github.com/UniversalDependencies/tools/tree/master/data/deprel.ud). Alternatively the new (json) format can be used (https://github.com/UniversalDependencies/tools/blob/master/data/deprels.json) together with the option --language
• --features <file> comma separated list of files, containing valid feature=value pairs (see https://github.com/UniversalDependencies/tools/tree/master/data/feat_val.ud) in addition to feature=value pairs, a second type of lines is possible to define the list of features which are valid for a given UPOS: for instance U:NOUN Gender Number Case Alternatively the new (json) format can be used (https://github.com/UniversalDependencies/tools/blob/master/data/feats.json) together with the option --language. This will highlight features which are not used with a correct UPOS and enable afeature edit mode in table view mode (see below)
• --language <lg code> use feature and/or deprel definitions in the json files given to the --features and --deprels options. Without --language only the universal features and deprels are used.
• --include_unused some features defined for a given language in feats.json are marked as unused. They will only be included to the list of valid features if this option is given.
• --validator <file> validator configuration file (see section validation below)
• --shortcuts <file> list of shortcut definitions (see section shortcuts below, format, cf. gui/multi-shortcuts.json)
• --shortcutTimeout <milliseconds> maximal tile allowed between to keys of a shortcut sequence
• --debug <hex> hex number to activate debug information of the server (printed to stderr)
• --saveAfter <number> if given, the server saves/commits the changed file only after number edits. To force saving the current state, click on the save button. Default: save when another sentence is chosen. This option can help to speed up the server when editing very large files, since writing the file after each edit takes a while, especially if the file is on a network drive.
• --relax accepts some formal errors in the CoNLL-U file and corrects them (empty column instead of _, invalid head id set to 0)
• --noedit deactivates editing, useful to browse an existing treebank and to avoid accidental errors.
• --reinit (implies --noedit) reloads the file at each navigation (in order to browse a file which is being modified by someone else)
• --rootdir root of fileserver (directory which includes index.html and edit.js etc. for ConlluEditor). Default: gui/ this option if only needed if the frontend is not in the default directory.

If the .conllu file contains major tree errors (like cycles, no token with head 0 or head ids beyond the end of the sentence, warnings are writte to screen. Such errors may occur if an automatic pre-annotation tools do not work correctly. UD parsers like Udpipe do not produce invalid files. In this case please correct the errors using a text editor before loading the file into ConlluEditor.

Editing

More help on editing can be found by clicking the Help button.

If the server has been (re)started reload the page in your navigator.

Load the first sentence by clicking on read sentence: clicking on a word and then clicking on the head-word creates a dependency relation. An edit window opens to enter the relation a name. Existing relations can be renamed by clicking on their name. Clicking twice on a word deletes its eventual dependency relation and makes it root. To edit form, lemma etc. CTRL-click o