github.com/Unstructured-IO/unstructured-api @0.1.2

API Announcement!

We are thrilled to announce our newly launched Unstructured API. While access to the hosted Unstructured API will remain free, API Keys are required to make requests. To prevent disruption, get yours here now and start using it today! Check out the readme here to get started making API calls.

:rocket: Beta Feature: Chipper Model

We are releasing the beta version of our Chipper model to deliver superior performance when processing high-resolution, complex documents. To start using the Chipper model in your API request, you can utilize the hi_res strategy. Please refer to the documentation here.

As the Chipper model is in beta version, we welcome feedback and suggestions. For those interested in testing the Chipper model, we encourage you to connect with us on Slack community.

---

General Pre-Processing Pipeline for Documents

This repo implements a pre-processing pipeline for the following documents. Currently, the pipeline is capable of recognizing the file type and choosing the relevant partition function to process the file.

Category	Document Types
Plaintext	.txt, .eml, .msg, .xml, .html, .md, .rst, .json, .rtf
Images	.jpeg, .png
Documents	.doc, .docx, .ppt, .pptx, .pdf, .odt, .epub, .csv, .tsv, .xlsx
Zipped	.gz

:rocket: Unstructured API

Try our hosted API! It's freely available to use with any of the filetypes listed above. This is the easiest way to get started. If you'd like to host your own version of the API, jump down to the Developer Quickstart Guide.

 curl -X 'POST' \
  'https://api.unstructured.io/general/v0/general' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -H 'unstructured-api-key: <YOUR API KEY>' \
  -F 'files=@sample-docs/family-day.eml' \
  | jq -C . | less -R

Parameters

Strategies

Four strategies are available for processing PDF/Images files: hi_res, fast, ocr_only and auto. fast is the default strategy and works well for documents that do not have text embedded in images.

On the other hand, hi_res is the better choice for PDFs that may have text within embedded images, or for achieving greater precision of element types in the response JSON. Please be aware that, as of writing, hi_res requests may take 20 times longer to process compared to the fast option. See the example below for making a hi_res request.

 curl -X 'POST' \
  'https://api.unstructured.io/general/v0/general' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'files=@sample-docs/layout-parser-paper.pdf' \
  -F 'strategy=hi_res' \
  | jq -C . | less -R

The ocr_only strategy runs the document through Tesseract for OCR. Currently, hi_res has difficulty ordering elements for documents with multiple columns. If you have a document with multiple columns that do not have extractable text, we recommend using the ocr_only strategy. Please be aware that ocr_only will fall back to another strategy if Tesseract is not available.

For the best of all worlds, auto will determine when a page can be extracted using fast or ocr_only mode, otherwise it will fall back to hi_res.

Hi Res model name

The hi_res strategy supports different models, and the default is detectron2onnx. You can also specify hi_res_model_name parameter to run hi_res strategy with the chipper model while using the host API:

 curl -X 'POST' \
  'https://api.unstructured.io/general/v0/general' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'files=@sample-docs/layout-parser-paper.pdf' \
  -F 'strategy=hi_res' \
  -F 'hi_res_model_name=chipper'  \
  | jq -C . | less -R

We also support models to be used locally, for example, yolox. Please refer to the using-the-api-locally section for more information on how to use the local API.

OCR languages

Note: This kwarg will eventually be deprecated. Please use languages. You can also specify what languages to use for OCR with the ocr_languages kwarg. See the Tesseract documentation for a full list of languages and install instructions. OCR is only applied if the text is not already available in the PDF document.

curl -X 'POST' \
  'https://api.unstructured.io/general/v0/general' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'files=@sample-docs/english-and-korean.png' \
  -F 'strategy=ocr_only' \
  -F 'ocr_languages=eng'  \
  -F 'ocr_languages=kor'  \
  | jq -C . | less -R

Languages

You can also specify what languages to use for OCR with the languages kwarg. See the Tesseract documentation for a full list of languages and install instructions. OCR is only applied if the text is not already available in the PDF document.

curl -X 'POST' \
  'https://api.unstructured.io/general/v0/general' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'files=@sample-docs/english-and-korean.png' \
  -F 'strategy=ocr_only' \
  -F 'languages=eng'  \
  -F 'languages=kor'  \
  | jq -C . | less -R

Coordinates

When elements are extracted from PDFs or images, it may be useful to get their bounding boxes as well. Set the coordinates parameter to true to add this field to the elements in the response.

 curl -X 'POST' \
  'https://api.unstructured.io/general/v0/general' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'files=@sample-docs/layout-parser-paper.pdf' \
  -F 'coordinates=true' \
  | jq -C . | less -R

Skip Table Extraction

Currently, we provide support for enabling and disabling table extraction for all file types. Set parameter skip_infer_table_types to specify the document types that you want to skip table extraction with. By default, we enable table extraction for all file types (skip_infer_table_types=[]). Again, please note that table extraction only works with hi_res strategy. For example, if you want to skip table extraction for images, you can pass a list with matching image file types:

 curl -X 'POST' \
  'https://api.unstructured.io/general/v0/general' \
  -H 'accept: application/json' \
  -H 'Content-Type: multipart/form-data' \
  -F 'files=@sample-docs/layout-parser-paper-with-table.jpg' \
  -F 'strategy=hi_res' \
  -F 'skip_infer_table_types=["jpg"]' \
  | jq -C . | less -R

Encoding

You can specify the encoding to use to decode the text input. If no value is provided, utf-8 will be used.

curl -X 'POST' \
 'https://api.unstructured.io/general/v0/general' \
 -H 'accept: application/json'  \
 -H 'Content-Type: multipart/form-data' \
 -F 'files=@sample-docs/fake-power-point.pptx' \
 -F 'encoding=utf_8' \
 | jq -C . | less -R

Gzipped files

You can send gzipped file and api will un-gzip it.

curl -X 'POST' \
 'https://api.unstructured.io/general/v0/general' \
 -H 'accept: application/json'  \
 -H 'Content-Type: multipart/form-data' \
 -F 'gz_uncompressed_content_type=application/pdf' \
 -F 'files=@sample-docs/layout-parser-paper.pdf.gz' 

If field gz_uncompressed_content_type is set, the API will use its value as content-type of all files after uncompressing the .gz files that are sent in single batch. If not set, the API will use various heuristics to detect the filetypes after uncompressing from .gz.

XML Tags

When processing XML documents, set the xml_keep_tags parameter to true to retain the XML tags in the output. If not specified, it will simply extract the text from within the tags.

curl -X 'POST' \
 'https://api.unstructured.io/general/v0/general' \
 -H 'accept: application/json'  \
 -H 'Content-Type: multipart/form-data' \
 -F 'files=@sample-docs/fake-xml.xml' \
 -F 'xml_keep_tags=true' \
 | jq -C . | less -R

Page Breaks

For supported filetypes, set the include_page_breaks parameter to true to include PageBreak elements in the output.

curl -X 'POST' \
 'https://api.unstructured.io/general/v0/general' \
 -H 'accept: application/json'  \
 -H 'Content-Type: multipart/form-data' \
 -F 'files=@sample-docs/layout-parser-paper-fast.pdf' \
 -F 'include_page_breaks=true' \
 | jq -C . | less -R

Unique element IDs

By default, the element ID is a SHA-256 hash of the element text. This is to ensure that the ID is deterministic. One downside is that the ID is not guaranteed to be unique. Different elements with the same text will have the same ID, and there could also be hash collisions. To use UUIDs in the output instead, set unique_element_ids=true. Note: this means that the element IDs will be random, so with every partition of the same file, you will get different IDs. This can be helpful if you'd like to use the IDs as a primary key in a database, for example.

curl -X 'POST' \ 
 'https://api.unstructured.io/general/v0/general' \
 -H 'accept: application/json'  \
 -H 'Content-Type: multipart/form-data' \
 -F 'files=@sample-docs/layout-parser-paper-fast.pdf' \
 -F 'unique_element_ids=true' \
 | jq -C . | less -R

Chunking Elements

Use the chunking_strategy form-field to chunk text into larger or smaller elements. Defaults to None which performs no chunking. The available chunking strategies are basic and by_title.

The basic strategy combines whole consecutive document elements to maximally fill chunks of max_characters length. A single element that by itself exceeds max_characters is divided into two or more chunks by text-splitting (on a word boundary).

The by_title strategy has the same behaviors except document section boundaries are respected, meaning elements from two different sections never occur in the same chunk. A Title (section heading) element introduces a new section, hence the name.

Additional Parameters (all optional):

`max_characters`
  The hard maximum chunk size. No chunk will exceed this length. Defa