Evaluation Metrics

Metric List

Since all tasks in General-Bench retain their original task definitions without altering the output or prediction format, our evaluation methods vary according to the nature of different tasks and data. Following, we show the evaluation metrics and methods used across all tasks.

General

Following, we show the general evaluation metrics.

Accuracy

Accuracy is defined as the ratio of correctly classified instances to the total number of instances.

Abbreviation:

Acc

Monotonicity:

monotonically increasing

Range:

[0,1]

Representative tasks:

Classification

Macro Accuracy

Macro-Acc evaluates how well a model performs on average across all classes, regardless of class imbalance

Abbreviation:

Macro-Acc

Monotonicity:

monotonically increasing

Range:

[0,1]

Representative tasks:

Event Relation Prediction

Ranking-related

Recall@k

R@k measures the Recall rate at the top k results in tasks like image retrieval, where the true positive must appear within the top k predicted results

Abbreviation:

R@k

Monotonicity:

monotonically increasing

Range:

[0,1]

Representative tasks:

Image Scene Graph Parsing

Regression-related

MAE

MAE, Mean Absolute Error, measures the average of the absolute differences between the predicted values and the actual values. It’s typically used in regression tasks.

Abbreviation:

MAE

Monotonicity:

monotonically decreasing

Range:

[0,∞]

Representative tasks:

Object Counting

Text Generation-related

BLEU-1

BLEU-1, Bilingual Evaluation Understudy (1-gram), calculates the precision of unigrams (individual words) in the generated text compared to the reference text(s).

Abbreviation:

BLEU-1

Monotonicity:

monotonically increasing

Range:

[0,1]

Representative tasks:

Text Generation

Image-related

PSNR

PSNR, Peak Signal-to-Noise Ratio, measures the quality of a reconstructed or compressed signal, such as images or videos, compared to the original signal.

Abbreviation:

PSNR

Monotonicity:

monotonically increasing

Range:

[0,∞]

Representative tasks:

Image Desnowing

Video-related

Frame-level Accuracy

Frame-level Accuracy evaluates how many frames (or time steps) in the sequence are correctly classified by comparing predictions with the ground truth on a frame-by-frame basis

Abbreviation:

Fram-Acc

Monotonicity:

monotonically increasing

Range:

[0,1]

Representative tasks:

Video Translation

Audio-related

Contrastive Language-Audio Pretraining Score

CLAP (Contrastive Language-Audio Pretraining) evaluates the alignment between generated audio and text. It is derived from a contrastive learning framework where embeddings of audio and text are trained to be close in a shared latent space if they are semantically related

Abbreviation:

CLAP

Monotonicity:

monotonically increasing

Range:

[0,1]

Representative tasks:

Audio Editing

3D-related

Average Multi-Object Tracking Accuracy

AMOTA (Average Multi-Object Tracking Accuracy) is a performance metric used to evaluate multi-object tracking (MOT) systems. It combines detection accuracy and tracking performance by assessing how well a system detects, associates, and tracks multiple objects over time. AMOTA is calculated by averaging tracking accuracy over a range of thresholds for the Intersection over Union (IoU) or matching criteria. :abbreviation: AMOTA :monotonicity: monotonically increasing :range: [0,1] :representative tasks: 3D Tracking

Segmentation&Detection-related

Mean Intersection over Union

mIoU (Mean Intersection over Union), calculates the ratio of the intersection area to the union area between the predicted and ground truth segmentation masks for a single class

Abbreviation:

mIoU

Monotonicity:

monotonically increasing

Range:

[0,1]

Representative tasks:

Image Semantic Segmentation

Human-aware Evaluation

GPT-Score

GPT-Score evaluates the instruction following rate with GPT assistance, as an alternative to human evaluation.

Abbreviation:

GPT-Score

Monotonicity:

monotonically increasing

Range:

[0,1]

Representative tasks:

Audio Question Answering

Mapping Functions of Scoring Metric

Most task evaluation scores, despite utilizing different metrics, fall within a 0-100% range, such as F1, Accuracy (Acc), and ROUGE-L, and follow a monotonically increasing trend. However, certain task metrics produce scores outside this range. For example, regression-related metrics, as well as FID, FVD, and similar metrics, range from 0 to infinity and follow a monotonically decreasing trend. In contrast, MOS scores are represented as a discrete 5-point scale. Due to these varying score ranges across tasks, it becomes intractable to normalize them to a unified scale for level score calculations. Thus, we design the following mapping functions to standardize these metrics into a 1-100% range, thereby streamlining the computation of level scoring algorithms.

• MAE

  \[y = 2 \times \text{sigmoid}\left(\frac{50}{x}\right) -1 , \quad \text{where } x \in [0, +\infty), \quad y \in (0, 1).\]

• RMS

  \[y = 2 \times \text{sigmoid}\left(\frac{50}{x}\right) -1, \quad \text{where } x \in [0, +\infty), \quad y \in (0, 1).\]