ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction

I submitted my results to the competition with the last model checkpoint. Unfortunately, my results are not reproducibles across different machines but were the same on my machine through multiple executions. Indeed, after having seeded the training, the results obtained were slightly different on the borrowed NVIDIA GTX 1080. This is not surprising because it is mentioned in the Pytorch documentation about reproducibility. That is why I gave up on seeding the training. One last important thing to note is this repository aims to be the base code for scientific researches on OCR.

Table of contents

1. Highlights
2. Challenge
   1. Overview
   2. Dataset and Annotations
      1. Description
      2. Mistakes
      3. Downloads
3. Methods
4. Results
5. User guide
   1. Software & hardware
   2. Conda environment setup
   3. Visualizer
6. Troubleshooting
7. References
8. Citations
9. TODO

Highlights

• Pytorch 1.7.0: Currently Pytorch 1.7.0 or higher is supported.
• Metrics visualization: Support Visdom for real-time loss visualization during training.
• Automatic mixed precision training: Train faster with AMP and less GPU memory on NVIDIA tensor cores
• Pretrained models provided: Load the pretrained weights.
• GPU/CPU support for inference: Runs on GPU or CPU in inference time.
• Intelligent training procedure: The state of the model, optimizer, learning rate scheduler and so on... You can stop your training and resume training exactly from the checkpoint.

Challenge

Overview

Scanned receipts OCR is a process of recognizing text from scanned structured and semi-structured receipts, and invoices in general. Indeed,

Scanned receipts OCR and information extraction (SROIE) play critical roles in streamlining document-intensive processes and office automation in many financial, accounting and taxation areas.

For further info, check the ICDAR19-RRC-SROIE competition.

Dataset and Annotations

Description

x1_1,y1_1,x2_1,y2_1,x3_1,y3_1,x4_1,y4_1,transcript_1

x1_2,y1_2,x2_2,y2_2,x3_2,y3_2,x4_2,y4_2,transcript_2

x1_3,y1_3,x2_3,y2_3,x3_3,y3_3,x4_3,y4_3,transcript_3

…

{
  "company": "STARBUCKS STORE #10208",
  "address": "11302 EUCLID AVENUE, CLEVELAND, OH (216) 229-0749",
  "date": "14/03/2015",
  "total": "4.95"
}

Mistakes

Another mistakes lie within the folder 0325updated.task2train(626p) and 0325updated.task1train(626p) which are used for the Task 3: Keyword Information Extraction. Indeed, there are files for which the date format, address and company are wrong and here are the corrections that were made:

Downloads

The Original dataset can be found Google Drive or Baidu NetDisk.

Taking into account to what was mentioned in dataset mistakes above, you may obviously want to make the changes by your own, but I have already made the corrections, and it can be downloaded via the bash script: sroie2019.sh and here is how to run it:

• Without specifying the directory, then it will download inside the default directory ~/SROIE2019:

  sh scripts/datasets/sroie2019.sh

• Specifying a new directory, let's say ~/dataset/ICDAR2019:

  sh scripts/datasets/sroie2019.sh ~/dataset/ICDAR2019

Do not forget to specify the new directory inside this file.

For Windows users who do not have bash on their system, you may want to install git bash. Once it is installed, you can set the entire git bin folder in the environment variables .

Methods

Results

User guide

Hardware

Conda environment setup

For Mac, Windows and Linux users, if conda is not installed, then you need to follow this documentation.

1. Updating conda

Before installation, we need to make sure conda is updated.

conda update conda

1. Creating an environment from a file

conda env create -f env/environment.yml

This will create a new conda environment named SROIE2019 on your system, which will give you all the packages needed for this repo. If you do not own any NVIDIA GPUs (with CUDA capable-system), then you must remove the cudatoolkit and cudnn lines in the environment.yml file. Otherwise, make sure your graphic card supports the installed version of CUDA.

1. Activating the new environment

conda activate SROIE2019

or

source activate SROIE2019

If you want to deactivate the environment, you can simply do: conda deactivate

1. Verify that the new environment was installed correctly

conda env list

or

conda info --envs

for further info, you can check the manager .

Visualizer

To use Vidsom, you must make sure the server is running before you run the training.

Starting the server with

python3 -m visdom.server

or simply

visdom

Visdom interface

In your browser, you can go to:

http://localhost:8097

You will see the visdom interface:

One important thing to remember...You can launch the server with a specific port. Let's say: visdom --port 8198. But you need to make sure the Visualizer runs with the port 8198.

For further info on Visdom, you can check this: https://github.com/fossasia/visdom

Troubleshooting

If you have issues running or compiling this code, there are a list of common issues in TROUBLESHOOTING.md. If your issue is not present there, then feel free to open a new issue.

References

• Luffic-SSD for the data augmentation and anchor matching parts.
• eragonruan, courao , tranbahien for the implementation of the CTPN used to tackle the text localization.
• eadst for the implementation for removing the extra white space in the scanned receipts for the text localization.
• HephaestusProject for the implementation of Character-Aware Neural Language Models used to tackle the keyword-information extraction.

Citations

@misc{blackstar1313_sroie_2019,
  author = {Njoyim Tchoubith Peguy Calusha},
  title  = {ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction},
  year   = {2021},
  url    = {https://github.com/BlackStar1313/ICDAR-2019-RRC-SROIE}
}

TODO