An AI quality-control test must count missed defects

Define the defect and the cost of missing it before testing AI for manufacturing quality control. Evaluate missed defects and false alarms separately. A single accuracy percentage can hide a failure that matters to the quality owner.

Begin with an offline evaluation using approved sample data and qualified reference judgments. The exercise should not replace an established inspection or release products. Its purpose is to learn whether a bounded model-assisted check deserves further validation.

Make the reference judgment repeatable

Choose one observable defect type and record the conditions under which it should be identified. Keep the product, revision, inspection method and acceptable variation explicit. If qualified inspectors disagree about the reference label, resolve that disagreement before using the sample to grade the model.

Keep the evaluation sample separate from data used to develop or tune the system. Include ordinary acceptable work and the relevant defect cases. Avoid claiming that a small, convenient sample represents every material, camera, lighting condition or production line.

Record an uncertain or unreadable case as such. Forcing every poor image into “good” or “bad” can hide the need for a retake or a human inspection.

Count the errors by their consequence

Consider this invented 100-item evaluation. Qualified reviewers identify ten defective items and 90 acceptable ones. The model flags eight of the ten defects and incorrectly flags nine acceptable items.

The model agrees with the reference on 89 of 100 items, or 89%. Yet it misses two of ten known defects, a 20% miss rate in this sample. Nine of 90 acceptable items are falsely flagged, a 10% false-alarm rate. Only eight of the 17 flags correspond to a known defect, about 47.1%.

Those calculations describe this invented sample only. They do not establish acceptable performance or a production defect rate. The quality owner must set the relevant acceptance criteria and determine the evidence required for the actual use.

Keep the model result separate from product disposition

A flag can create a review task. It should not silently become a final release or rejection decision unless that automation has been separately validated and authorized for the specific process.

Odoo 19.0 documents quality checks tied to manufacturing or inventory operations, with types including measurements, pass/fail and photographs. Its documentation describes inspection records; it does not establish a built-in AI vision capability or validate an external model. See Odoo's quality-check reference.

If you connect a model, define the record it may create, the evidence it attaches and the person who decides the disposition. Preserve the original observation and the reviewed result. The AI access worksheet helps limit the connection's permissions.

Test changes that can invalidate the result

Repeat the relevant evaluation when the product, revision, camera position, lighting, material appearance or model changes. Compare errors by the condition that produced them. An overall result can improve while one important defect type gets worse.

The NIST AI Risk Management Framework provides a voluntary basis for considering AI risk through design, use and evaluation. This worksheet is a proposed operating test, not certification or a substitute for the quality standards applicable to your product.

Measure the work created by the system too: retakes, false-alarm reviews, disputed labels and time spent investigating missed defects. The AI pilot-results guide keeps that effort in the business evaluation.

Before widening the pilot, ask the quality owner to explain every cell of the table and the disposition of uncertain cases. If the only available answer is an accuracy headline, the evaluation is not yet detailed enough for a release decision.

How we research and review these articles