Datasets & Annotation

Principles and Suitability

Annotation turns a task definition into reference answers, such as bounding boxes, masks, keypoints or numeric values. Its quality depends on annotation guidance and agreement between reviewers, not just the number of images collected.

Labels are the answers describing what a system should learn. Boxes locate objects approximately, masks identify their pixels, and keypoints locate important positions such as pointer pivots. These answer different questions. Dial boxes do not constitute complete reference answers for reading values. Choose labels according to the required output and the evidence actually available in each image, rather than assuming that more detailed labels always provide more useful information.

Guidance should define coordinate systems, object boundaries and invisible states, with accepted and rejected examples. Check which image version each answer belongs to: rotation and resizing change coordinates. Editor history and decision reasons distinguish correction of a wrong answer from a changed task definition. The latter can require systematic review of the entire dataset, not just adjustment of one disputed example.

Scroll the diagram horizontally to inspect all details.

Illustrative Industrial Example

Illustrative design scenario: label a gauge with its dial boundary, pivot, pointer tip and reading from a reference source. Mark a point as not visible rather than guessing where the tip lies beneath a reflection.

Suppose the pointer base is clear but glare hides its tip. Mark visible parts and the tip's unknown status instead of placing it where an expected reading suggests. If a reference instrument supplies a value, distinguish that information from visible evidence. Do not penalise a model for failing to read details the camera never captured, or present the externally supplied value as something directly observed in the image.

Have reviewers label a pilot independently before comparing disagreements, such as whether a box includes the metal rim or only the scale. Once definitions are agreed, revise guidance and review earlier work. Check answer files for in-image coordinates, valid class names and unique identifiers. Export genuine zero distinctly from empty or unknown values so missing annotations cannot become apparently valid numerical readings.

Study and Trial Workflow

Evaluation and Metric Definitions

Measure agreement using box IoU, keypoint pixel distance or reading differences, according to the label type. Agreement between annotators does not independently confirm the accuracy of the reference instrument.

IoU compares intersection and union areas, describing box boundaries but not proving pointer-tip correctness. Interpret point distances relative to dial size: the same pixel difference can matter differently on large and small images. Compare numerical answers in identical units with reference times aligned to capture. Reviewers agreeing on an image still cannot establish a reference instrument's accuracy without separate supporting evidence.

Sample apparently easy and borderline images, not only cases the model suggests are wrong. Report differences by label type and reviewer, with unresolved-case counts. If boxes agree but readings differ substantially, inspect units, scales and reference times before calling all labels good. Agreement measurements improve the annotation process; they do not certify every reference answer as true.

Limitations and Failure Cases

Labels copied from model predictions without review may amplify existing errors. One image may also be insufficient to establish the true reading of a gauge affected by parallax, even if the annotation looks precise.

Automatic proposals can reduce drawing work, but reviewers may accept them without noticing what the model missed. Include full-image reviews and examples annotated from scratch to investigate tool-induced bias. Images without target objects are useful negative evidence. Do not delete them merely because there are no boxes to draw: they help define what should not be labelled as a gauge.

Detailed labels cost time and create more opportunities for disagreement. Use the detail the downstream task needs, not elaborate masks merely to make a dataset look complete. Retain unknown statuses and reasons when images cannot establish answers. Review image rights and identifying information before publication or transfer; annotation does not make an image free of its original usage restrictions.

Practical Checklist

• Are there examples defining the acceptance boundaries?
• Is an unknown reading distinguished from a numeric zero?
• Have image usage rights and personally identifiable data been reviewed?

References and Further Reading

• Gebru et al. — Datasheets for Datasets
• COCO — Detection evaluation