Metric: isco_hierarchical_accuracy

ISCO-08 sample-paired hierarchical precision, recall and F1, based on Kiritchenko, Matwin and Famili (2005). Each code is extended with its ancestors, excluding the universal root. Overlap is calculated within each aligned sample, then counts are summed across samples. Every hierarchy level has equal weight.

ISCO codes must be wrapped in double quotes.

Metric Card for ISCO-08 Hierarchical Accuracy Measure

Metric Description

ISCO‑08 is a four‑level taxonomy (Table 1). Correctly locating an occupation at the appropriate level—Major, Sub‑major, Minor or Unit—is essential, yet many practical systems make small but still useful errors (e.g. confusing two Unit groups within the same Minor group). The present metric extends classical precision, recall and F‑measure to this hierarchical setting, thereby providing a more nuanced assessment than flat accuracy.

In this context, the hierarchical accuracy measure is specifically designed to evaluate classifications within this structured framework. It emphasizes the importance of precision in classifying occupations at the correct level of specificity:

1. Major Groups (1-digit codes): The broadest category that encompasses a wide range of occupations grouped based on their fundamental characteristic and role in the job market. Misclassification among different Major groups implies a fundamental misunderstanding of the occupational role.

2. Sub-major Groups (2-digit codes): Provide a more detailed classification within each Major group, delineating categories that share specific occupational characteristics. Errors within a Major group but across different Sub-major groups are less severe than those across Major groups but are still significant due to the deviation from the correct occupational category.

3. Minor Groups (3-digit codes): Offer further granularity within Sub-major groups, classifying occupations that require similar skill sets and qualifications. Misclassifications within a Sub-major group but across different Minor groups are penalized, yet to a lesser extent than errors across Sub-major or Major groups, as they still fall within a broader correct occupational category.

4. Unit Groups (4-digit codes): The most specific level, identifying precise occupations. Misclassification within a Minor group, between different Unit groups, represents the least severe error, as it occurs within the context of closely related occupations.

The hierarchical accuracy measure rewards more precise classifications that correctly identify an occupation's placement down to the specific Unit group level and applies penalties for misclassifications based on the hierarchical distance between the correct and assigned categories. This approach allows for a refined evaluation of classification systems, acknowledging partially correct classifications while penalizing inaccuracies more severely as they deviate further from the correct level in the hierarchical structure.

For example, a misclassification into Unit group 2211 (Generalist Medical Practitioners) when the correct category is Unit group 2212 (Specialist Medical Practitioners) should incur a lesser penalty than misclassification into Unit group 2352 (Special Needs Teachers), as Unit groups 2211 and 2212 are within the same Minor group 221 (Medical doctors) and share a closer hierarchical relationship compared to Minor group 235 (Other teaching professionals) which is in a different Sub-major group (23 Teaching professionals).

The measure applies a higher penalty for errors that occur between more distant categories within the hierarchy:

1. Correct classifications at a more specific level (e.g., Unit group 2211) are evaluated more favorably than classifications at a more general level within the same hierarchy (e.g., Sub-major group 22), since the former indicates a closer match to the correct category.
2. Conversely, incorrect classifications at a more specific level are penalized more heavily than those at a more general level. For instance, classifying into Minor group 222 (Nursing and Midwifery Professionals) is considered worse than classifying into its parent category, Sub-major group 22, if the correct classification is Minor group 221, because the incorrect specific classification indicates a farther deviation from the correct category.

Misclassification among sibling categories (e.g., between different Minor groups within the same Sub-major group) is less severe than misclassification at a higher hierarchical level (e.g., between different Major groups).

The measure extends the concepts of precision and recall into a hierarchical context, introducing hierarchical precision (ℎ𝑃) and hierarchical recall (ℎ𝑅). In this framework, each sample belongs not only to its designated class but also to all ancestor categories in the hierarchy, excluding the root (we exclude the root of the graph, since all samples belong to the root by default). This adjustment allows the measure to account for the hierarchical structure of the classification scheme, rewarding more accurate location of a sample within the hierarchy and penalizing errors based on their hierarchical significance.

To calculate the hierarchical measure, extend the set of real classes

$$C_i = \{G\}$$

with all ancestors of 𝐺:

$$\vec{C}_i = \{B, C, E, G\}$$

Similarly, extend the set of predicted classes

$$C^′_i = \{F\}$$

with all ancestors of 𝐹:

$$\vec{C}^′_i = \{C, F\}$$

Class 𝐶 is the only correctly assigned label from the extended sets:

$$| \vec{C}_i ∩ \vec{C}^′_i| = 1$$

There are

$$| \vec{C}^′_i| = 2$$

predicted labels and

$$| \vec{C}_i| = 4$$

real classes.

Therefore,

$$hP = \frac{| \vec{C}_i ∩ \vec{C}^′_i|} {|\vec{C}^′_i |} = \frac{1}{2}$$

$$hR = \frac{| \vec{C}_i ∩ \vec{C}^′_i|} {|\vec{C}_i |} = \frac{1}{4}$$

Finally, combine ℎ𝑃 and ℎ𝑅 into the hierarchical 𝐹-measure:

$$hF_β = \frac{(β^2 + 1) · hP · hR}{(β^2 · hP + hR)}, β ∈ [0, +∞)$$

The implementation returns hF₁, with β fixed at 1. Across samples, it computes micro-averaged precision and recall by summing within-sample path intersections:

$$hP = \frac{\sum_i |T_i \cap P_i|}{\sum_i |P_i|}, \qquad hR = \frac{\sum_i |T_i \cap P_i|}{\sum_i |T_i|}.$$

Here, $T_i$ and $P_i$ contain the reference or predicted code and all its ancestors, excluding the universal root. Every node has equal weight. Labels are never pooled across different samples, and repeated records contribute separately. This follows Kiritchenko, Matwin and Famili (2005), section 3.3.

When both inputs contain only four-digit unit codes, every path contains four nodes, so hP = hR = hF₁ = the mean accuracy at the 1-, 2-, 3- and 4-digit levels. A wrong unit code earns 0, 0.25, 0.50 or 0.75 according to the number of shared ancestor levels; an exact match earns 1. This assigns equal credit to each matched hierarchy level. With partial codes, path sizes differ; the implementation aggregates node counts before calculating F1 rather than averaging per-sample F1 scores.

The metric rewards depth: predicting 221 instead of 22 yields higher hF because more ancestors overlap. It penalises distance: predicting a Unit in a different Sub‑major group incurs a larger loss than confusing two Units under the same Minor group.

How to Use

• Model evaluation Assess neural or rule‑based classifiers that map free‑text occupation descriptions to ISCO‑08 codes.
• Inter‑rater agreement Quantify consistency between human coders when full agreement at the Unit level is not always expected.

Inputs

• references (List[str]): List of reference ISCO-08 codes (true labels). This is the ground truth.
• predictions (List[str]): List of predicted ISCO-08 codes (predicted labels). This is the predicted classification or classification to compare against the ground truth.

Inputs must be aligned, nonempty, and equally sized, with one valid code per sample. Codes at all four levels are accepted. A code may be followed by whitespace and a description, such as "1111 Legislators"; each label is normalized independently. Use strings to preserve leading zeros ("0110"). Unknown codes, missing values, and special codes such as "9998" and "9999" raise ValueError; handle them before evaluation and report any exclusions. The multilabel configuration is unsupported.

Output Values

Example (using the corrected local implementation):

import evaluate

metric = evaluate.load("./isco_hierarchical_accuracy.py")
metric.compute(references=["1111"], predictions=["1112"])
# Returns:
{
    "accuracy": 0.0,
    "hierarchical_precision": 0.75,
    "hierarchical_recall": 0.75,
    "hierarchical_fmeasure": 0.75,
}

Values are decimal numbers between 0 and 1. Higher scores are better. For references=["1111", "2111"] and predictions=["2111", "1111"], all four scores are 0: swapping codes across samples cannot earn credit from other records.

Legacy results

The following table is from the paper Improving Parental Occupation Coding Procedures with AI.

These are historical reported values, not benchmarks for the corrected metric. The earlier implementation pooled distinct codes and ancestors across the whole evaluation group, discarding sample pairing and frequencies. Its hierarchical scores are not comparable to the corrected sample-paired hF₁ and must be recomputed from the original paired predictions and references before interpreting classification performance. Existing result JSON files and saved notebook outputs have not been recomputed. Exact accuracy used paired records, although its old label parsing also needs checking when descriptive labels were mixed with bare codes.

Summary of multilingual model overall accuracies and hF-measure scores

Examples

def compute_metrics(p: EvalPrediction):
    preds = p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions
    preds = np.argmax(preds, axis=1)
    # Use labels instead of ids for hierarchical ISCO-08 classification
    preds = [model.config.id2label[p] for p in preds]
    refs = [model.config.id2label[r] for r in p.label_ids]
    result = metric.compute(predictions=preds, references=refs)
    return result

Limitations and Bias

Hierarchy overlap measures agreement in the taxonomy, not the consequences of an error for socioeconomic scores or other downstream uses. Equal credit per hierarchy level is a modelling assumption. The score gives each sample equal influence when both inputs are unit codes; frequent occupations can therefore dominate the result. Report exact unit accuracy and accuracy at each hierarchy level alongside hF₁. For human coders, this is raw hierarchical agreement, not chance-corrected reliability.

Verification

With the dependencies installed, run python -m unittest -v test_metric. Tests cover cross-sample swaps, repeated records, partial codes, normalization, leading zeros, invalid inputs, and the level-accuracy identity across all 436 unit codes. Integration tests load the local Hugging Face metric and check compute and add_batch, using the bundled taxonomy without network access.

Citation

This metric was developed as part of an IEA R&D project, Improving Parental Occupation Coding Procedures with AI (Duckworth et al, 2024). The metric was used to evaluate the multilingual ICILS XLM-R ISCO classification model.

Further References

• The International Standard Classification of Occupations- ISCO-08
• Improving Parental Occupation Coding Procedures with AI (Duckworth et al, 2024)
• ICILS XLM-R ISCO model