Skip to main content

Dev Station Technology

AI Defect Detection: What Data You Need and What Accuracy Hides

AI defect detection finds faults in parts from images, using one of three methods. How much training data you need is set by how rare each defect is, and a single accuracy percentage hides the two numbers an operator actually lives with.

Key takeaways

  • Data: How much you need is set by how rare the defect is, not by the model.
  • Methods: 3 methods cover almost every case, and only supervised learning needs labelled defect examples.
  • Accuracy: A model scoring 98.8 percent can still miss 1 defect in 5.
  • Drift: Lamp ageing, a new supplier finish or a moved camera all break a signed off model.
  • Dev Station: Dev Station builds defect detection systems for manufacturers in the United States and the United Kingdom.

01 / 07

Why do published answers about data and accuracy disagree?

Published answers disagree because every writer is describing a different defect. Some vendors say a few dozen images per defect class is enough to start, others put the minimum in the hundreds, and others describe production models holding tens of thousands of images.

Every AI defect detection conversation arrives at the same two questions. How much data do we need, and how accurate is it.

The spread exists because the number is set by your defects rather than by the technique. Both questions have precise answers once you replace them with better ones: how often does each defect occur, and what does a wrong decision cost in each direction.


02 / 07

What methods does AI defect detection use?

Three methods cover almost every industrial case: rule based vision, supervised learning and anomaly detection. The method follows from what you can show the model, and that is the whole decision.

Three defect detection methods compared: classical rule-based vision for measurement, supervised learning for known defect types, and anomaly detection trained only on good parts for rare or unseen defects
What you can show the model decides the method. Only the middle column needs defect examples.

Classical rule based vision measures. Is the hole 4.2mm, is the label present, is the gap within tolerance, are there sixteen pins. No training set, no drift, and a result you can explain to an auditor in one sentence. A surprising share of inspection problems brought to us as machine learning problems are this, and solving them this way is faster and cheaper.

Supervised learning needs labelled examples of each condition you want flagged. Classification says which defect, detection says where it is, and segmentation traces its exact outline, in that order of data appetite. Use it when the defect types are known, repeat, and matter enough to name.

Anomaly detection is trained on good parts only and flags anything that does not look like them. Rare, varied or not yet invented defects are what it is for, which describes most real production lines after the first month. The trade is that it tells you something is odd without telling you what, so a person still classifies the flag.

The common mistake: choosing supervised learning because it sounds more capable, then spending nine months waiting for enough examples of a defect that happens twice a quarter. Anomaly detection starts with the good parts you already make every day.


03 / 07

How much data do you need for AI defect detection?

Work it out from your own defect rate rather than from a published figure. Suppose a defect appears on one part in five hundred, and your model needs three hundred examples of it to learn the shape reliably.

Collecting those naturally means inspecting one hundred and fifty thousand parts. At a thousand parts a day that is five months of production before training can even begin, and that is for one defect class.

Three levers move that timeline, and none of them touches the model. You can narrow the problem to the defect classes that carry real cost, you can manufacture defective samples deliberately instead of waiting for them, or you can switch to anomaly detection and need no defect examples at all. Most programmes end up doing two of the three.

Two further points change the number. Segmentation needs considerably more labelled data than detection, because somebody has to trace outlines rather than draw boxes, so ask whether you need the outline before paying for it. And every variant counts separately: the same scratch on a matt surface and a polished one is two learning problems unless the lighting is controlled hard enough to make them look alike.


04 / 07

What does accuracy actually mean for a defect detection model?

Accuracy is the share of all decisions that were right, and on a line where defects are rare it is almost meaningless. A model that approves everything is already 99 percent accurate on a 1 percent defect rate.

Worked example on ten thousand parts with a one percent defect rate: a model catching eighty of one hundred defects and falsely flagging ninety nine good parts is 98.8 percent accurate, with 45 percent precision and 80 percent recall
The same model, described three ways. Only two of the three tell an operator what to expect.

Work it through on ten thousand parts with one hundred defects among them. A model that catches eighty of the hundred and also wrongly flags one percent of the good parts throws ninety nine false alarms. Its accuracy is 98.8 percent, which reads well in a proposal.

On the line that means two things. Twenty defects reached the customer, and of the one hundred and seventy nine parts pulled for review, only eighty had anything wrong with them. The first number is recall at 80 percent and the second is precision at 45 percent, and those are the two to put in a contract.

Which one you favour is a business decision rather than a technical one. A missed defect in an injectable medicine is not comparable to a missed cosmetic blemish on a bracket, and the threshold that trades recall against precision should be set by whoever owns that consequence. Ask any vendor quoting a single accuracy figure for the confusion matrix behind it, on your parts.


05 / 07

Why does a signed off model stop working?

Models do not decay, lines do, and the model notices before anyone else. The usual causes are physical rather than statistical.

A lamp ages and the image gets warmer. A fixture is replaced and the part sits two millimetres over, a new supplier delivers the same component with a different surface finish, somebody cleans the lens, somebody moves the camera to get at a jam.

Each change shifts the images away from what the model was trained on. The first symptom is usually a rise in false alarms that operators quietly start ignoring.

The defence is unglamorous. Keep a fixed set of golden samples and run them daily, record every image the model judged along with the decision and any human override, and review the overrides weekly. That review set is also your next training set, which is the part teams skip and then regret, as covered in the AI governance framework.


06 / 07

How do you write the acceptance test before buying?

Write it in five parts, before the first demonstration rather than after it. The acceptance test is the document that decides whether the project succeeds.

  1. List the defect classes and how often each occurs. Pull it from your own reject records. The rare ones set your timeline and the common ones set your return.
  2. State the cost of each error direction. What a missed defect costs you, and what a false alarm costs in pulled parts and operator time. Without both, nobody can set the threshold.
  3. Set recall and precision targets per class. Not one accuracy number for the whole line. Critical and cosmetic defects deserve different thresholds.
  4. Hold back a test set nobody trains on. Parts from a different week, a different shift and a different batch. A model evaluated on the data it learned from tells you nothing.
  5. Agree who retrains it and how often. Name a person. The readiness questions behind this sit in the AI readiness assessment.

A problem that resists this document is not ready for a model. Deciding between the broader options first is covered in automated visual inspection, and the question of whether to do it at all in when AI visual inspection is worth it. The wider workflow view sits in what AI inspection actually covers.


07 / 07

What has to be built around the model?

The loop around the model takes months, while the model itself takes weeks. That loop is what decides whether the model is still right next year.

Dev Station builds the whole of it: capture with controlled lighting and consistent framing, the inference service, the operator screen where a flag gets accepted or overruled, and the record that keeps the image, the model version, the suggestion and the human decision together.

The record is what lets you retrain on reality instead of on the original sample, and it is what an auditor asks for. The component view of the hardware side is in machine vision quality control, and the application survey in computer vision applications.

Most of our work is for manufacturers and inspection operators in the United States and the United Kingdom, so SOC 2 on one side and GDPR with ISO 27001 on the other sit in the specification rather than in a later remediation. Our engineers work from Vietnam with overlap into US Eastern, US Pacific and UK GMT hours, and we invoice in USD or GBP.

Send us your defect list and your reject rates and we will tell you which classes are worth a model this year. The build side sits in AI and ML development.


Frequently asked questions

Quality and operations teams ask these before scoping a vision project.

How many images do you need for AI defect detection?

The number depends on how rare the defect is and whether you need classification, detection or segmentation. Work out how many parts you must inspect to collect the examples naturally, because that calculation sets the timeline, and if the answer runs into months, use anomaly detection trained on good parts instead.

What is the difference between defect detection and anomaly detection?

Defect detection in the supervised sense learns from labelled examples of each defect you want found. Anomaly detection learns what a good part looks like and flags anything different, so it needs no defect examples and catches conditions nobody anticipated, at the cost of not naming what it found.

Is 99 percent accuracy good?

A single accuracy figure means nothing on its own. If one part in a hundred is defective, a model that approves everything scores 99 percent, so ask for recall and precision per defect class, measured on parts the model never trained on.

Why did accuracy drop after installation?

Physical conditions moved, almost always. Lamp ageing, a replaced fixture, a new supplier finish or a nudged camera all shift the images away from the training set, which is why daily golden sample checks and a retraining owner belong in the original scope.

Can we start without any defective samples?

Yes, with anomaly detection or with classical measurement rules where the check is dimensional. Defective examples are still worth collecting later, to tell the operator what kind of problem a flag represents.

Ask an AI about this

Want an AI assistant to summarize or cite this guide?

Click any link below to open the AI with a pre-filled prompt referencing this article:

Talk To Us

Tell us what you are building and what it has to connect to. An engineer answers, and you get a straight view of what the work would take.

Get In Touch →

Related articles

Let's Talk