Metrics

Built-in metrics

A large number of classic classification metrics is defined in the package:

• binary_accuracy
• precision
• recall
• sensitivity
• fall_out
• miss_rate
• specificity
• jaccard
• support
• true_false_ratio
• Fβ_score

All metrics can be applied to four scalars TP, FN, FP and TN.

using ClassificationMetrics

recall(25, 0, 1, 1)

1.0

If those arguments are not scalars but vectors, or ConfusionMatrix, PredictionResults, or predicted and actual labels, then the aggregation strategy will be used.

actual    = [0, 0, 0, 1, 1, 1, 2, 2, 2]
predicted = [1, 0, 0, 0, 0, 1, 0, 2, 0]

recall(predicted, actual; aggregation=macro_aggregation)

0.4444444444444444

It is also possible to specify no_aggregation as the aggregation strategy to return values for each class. By default, it uses aggregation created by the default_aggregation function. The default can be changed by calling set_default_aggregation!:

ClassificationMetrics.set_default_aggregation!(no_aggregation)

recall(predicted, actual, sort_labels=true)

3-element Vector{Float64}:
 0.6666666666666666
 0.3333333333333333
 0.3333333333333333

Aggregation

There are 4 standard aggregation strategies available:

• no_aggregation
• micro_aggregation
• macro_aggregation
• weighted_aggregation

By default, weighted_aggregation uses class supports to calculate weights, but it is possible to pass custom class weights as an optional keyword argument. For convenience of passing the function as the argument, it is possible to call weighted_aggregation with 1 argument to create a function with custom weights:

recall(predicted, actual, sort_labels=true, aggregation=weighted_aggregation([2, 1, 1]))

0.49999999999999994

Custom aggregation can be used by defining your own function:

using Statistics

micro_median_aggregation(func, TP, FN, FP, TN) = median(func.(TP, FN, FP, TN))

recall(predicted, actual, sort_labels=true, aggregation=micro_median_aggregation)

0.3333333333333333

Custom metrics

Defining a custom metric is also made easy thanks to the @metric macro. In order to define a metric, one needs only to define a function that calculates it for scalar TP, FN, FP and TN, and the @metric macro will automatically define all other functions, allowing the metric to be called for ConfusionMatrix, PredictionResults, and predicted/actual labels. It also allows to optionally define a display name for a metric that will be used in the classification_report:

using ClassificationMetrics: safe_div

# Option 1: define a function and then extend it in a separate macro call

true_negative_rate(TP, FN, FP, TN) = 1 − false_positive_rate(TP, FN, FP, TN)

@metric true_negative_rate

# Option 2: use macro on the function definition

@metric "Bookmaker informedness" informedness(TP, FN, FP, TN) = true_positive_rate(TP, FN, FP, TN) + true_negative_rate(TP, FN, FP, TN) − 1

classification_report(predicted, actual, sort_labels=true, metrics=[true_negative_rate, informedness])

┌──────────┬────────────────────┬────────────────────────┬─────────┐
│          │ True negative rate │ Bookmaker informedness │ Support │
├──────────┼────────────────────┼────────────────────────┼─────────┤
│        0 │             0.3333 │                 0.0000 │       3 │
│        1 │             0.8333 │                 0.1667 │       3 │
│        2 │             1.0000 │                 0.3333 │       3 │
│    Micro │             0.7222 │                 0.2500 │       9 │
│    Macro │             0.7222 │                 0.1667 │       9 │
│ Weighted │             0.7222 │                 0.1667 │       9 │
└──────────┴────────────────────┴────────────────────────┴─────────┘

Note that a custom name can be assigned to the metric by passing a string before the function name. That name will be used in the classification_report.