Invoke `vlm_eval.util.evaluation.gqa --> eval.py`; capture output to parse for accuracy/metrics.

r"""Compute the cosine similarity of two vectors.

Given a PIL.Image, pad to square by adding a symmetric border around the height/width.

Return (question_id: int, question: str, img_path: Path, answer: str) for an example.

Return (question_id: int, question_prompt: str, pixel_values: torch.Tensor, question: str, answer: str).

Return (example_id: int, caption: str, img_path: Path, true_false: bool) for an example.

Return (example_id: int, caption_prompt: str, pixel_values: torch.Tensor, caption: str, true_false: bool).

Return (question_id: int, question: str, img_path: Path, answer: int) for an example.

Return (qid: int, qprompt: str, pixel_values: torch.Tensor, question: str, answer: int).

Return (example_id: int, ref_expression: str, img_path: Path, bbox: List[float]) for an example.

Return (example_id: int, ref_expr_prompt: str, pixel_values: Tensor, ref_expr: str, bbox: List[float]).

Return (question_id: int, question: str, img_path: Path, answer: int) for an example.

Return (qid: int, qprompt: str, pixel_values: torch.Tensor, question: str, answer: int).

Return (question_id: int, question: int, img_path: Path, answer: str) for an example.

Return (qid: int, qprompt_ocr: str, qprompt_no_ocr: str, pixel_values: torch.Tensor, q: str, ans: str).

Return (question_id: int, question: int, img_path: Path, answer: str) for an example.

Return (question_id: int, question_prompt: str, pixel_values: torch.Tensor, question: str, answer: str).

Return (question_id: int, question: int, img_path: Path, answer: str) for an example.

Return (question_id: int, question_prompt: str, pixel_values: torch.Tensor, question: str, answer: str).

Return (question_id: int, question: int, img_path: Path, answer: str) for an example.

Return (question_id: int, question_prompt: str, pixel_values: torch.Tensor, question: str, answer: str).

Return (example_id: int, ref_expression: str, img_path: Path, bbox: List[float]) for an example.

Return (example_id: int, ref_expr_prompt: str, pixel_values: Tensor, ref_expr: str, bbox: List[float]).

Initializer for an Overwatch object that wraps logging & `accelerate.PartialState`.

Initializer for an Overwatch object that just wraps logging.

Constructs a lightweight PyTorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the GQA Test-Dev Set. In addition to the path to the

Task Runner for the GQA Dataset; loads data, then runs (distributed) VLM evaluation and writes results.

Wrapper around the official GQA evaluation script; handles converting results to/from GQA format.

Constructs a lightweight PyTorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the VSR Zero-Shot Test Set. In addition to the path

Task Runner for the VSR Dataset; loads data, then runs (distributed) VLM evaluation and writes results.

Computes exact-match True/False accuracy.

Constructs a lightweight Pytorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the AI2D Test Set. In addition to the path to the da

Task Runner for the AI2D Dataset; loads data then runs (distributed) VLM evaluation and writes results.

Computes exact-match multiple choice accuracy.

Constructs a lightweight PyTorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the RefCOCO Validation Sets. In addition to the path

Task Runner for the RefCOCO Dataset; loads data, then runs (distributed) VLM evaluation & writes results.

Computes Acc @ 0.5 IOU --> standard RefCOCO / RefCOCO+ / RefCOCOg metric.

Constructs a lightweight Pytorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the TallyQA Test Set. In addition to the path to the

Task Runner for the TallyQA Dataset; loads data then runs (distributed) VLM evaluation and writes results.

Computes exact-match multiple choice accuracy.

Constructs a lightweight PyTorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the Text VQA Validation Set. In addition to the path

Task Runner for the Text VQA Dataset; loads data, then runs (distributed) VLM evaluation and writes results.

Wrapper around the official Text VQA evaluation script; handles converting results to/from Text VQA format.

Constructs a lightweight PyTorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the VQAv2 Validation Set. In addition to the path to

Task Runner for the VQAv2 Dataset; loads data, then runs (distributed) VLM evaluation and writes results.

Wrapper around the official VQAv2 evaluation script; handles converting results to/from VQAv2 format.

Constructs a lightweight PyTorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the Pope Val Set. In addition to the path to the dat

Task Runner for the Pope Dataset; loads data, then runs (distributed) VLM evaluation and writes results.

Wrapper around the official Pope evaluation script; handles converting results to/from Pope format.

Constructs a lightweight PyTorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the VizWiz Val Set. In addition to the path to the d

Task Runner for the VizWiz Dataset; loads data, then runs (distributed) VLM evaluation and writes results.

Wrapper around the official VizWiz evaluation script; handles converting results to/from VizWiz format.

Constructs a lightweight PyTorch Dataset that loads from an index file and just returns metadata.

Constructs a fully-fledged PyTorch Map-Style Dataset for evaluating on splits of the OCID-Ref Validation Sets. In addition to the pat

Task Runner for the OCID-Ref Dataset; loads data, then runs (distributed) VLM evaluation & writes results.

Computes Acc @ 0.25 IOU --> standard metric for OCID-Ref splits (Min/Med/Max Clutter).

Provide `data_root` folder which contains `refclef`, `refcoco`, `refcoco+`, and `refcocog`. Additionally provide `dataset` name and `