Explainable AI (XAI)

Principles and Suitability

Grad-CAM highlights regions associated with class scores, while SHAP and LIME estimate feature contributions under their respective assumptions. These tools support model inspection but do not establish causality or answer correctness. A plausible visualization can help formulate a question about behavior; it cannot by itself show that the model understood a physical process or made a valid reading for the intended task.

A heatmap uses colors to show method-defined quantities, not human-like understanding of what a model sees. An overlay adds information such as a needle direction, which may support more direct checking of geometric formulas. Choose explanations according to the user's question rather than treating reasonable-looking colors as proof of accuracy or a complete account of physical causes. Different explanations answer different inspection needs and must retain their limits.

Engineers specify the Grad-CAM layer or a feature-method comparison baseline. The baseline is a reference condition whose selection affects explanations. SHAP estimates contributions according to its definition; LIME fits a local approximation near the case of interest. Both assume particular ways of changing inputs. Check stability and do not interpret contribution values as causal effects in the production process without additional evidence supporting that interpretation.

Scroll the diagram horizontally to inspect all details.

Illustrative Industrial Example

Design example: use a heatmap to investigate whether a gauge-type classifier relies on a logo instead of the dial, then test logo masking in an analysis collection. Separately, overlay the pivot and needle direction for a geometric reader. The classifier visualization investigates a hypothesis about learned dependence, while the geometric overlay supports tracing actual points, angles, and formulas in the reading path.

A map emphasizing a logo does not prove that the model relies only on that logo. Test the hypothesis with appropriately altered analysis data and inspect actual output changes. Unnatural masking can introduce a new disturbance, making a changed prediction inconclusive about the original hypothesis. Reviewers should see before-and-after images and the experiment's limitations rather than receive only a colored picture and a definitive causal story.

Engineers compare explanations with verified failure cases and similar successful images. For geometric readers, show the pivot, tip, and angle to support formula review; do not replace units and scale information with a heatmap. Store the method, layer, baseline, and model version together. Explanation changes after configuration changes may reflect the explanatory procedure rather than genuine changes in what the model uses for its decision.

Study and Trial Workflow

Evaluation and Metric Definitions

Measure stability and agreement with controlled input changes, and test whether explanations help users find real mistakes rather than merely increase confidence. An explanation should be evaluated as a review aid with its own assumptions. User satisfaction or a visually convincing map does not establish that the reviewer can distinguish wrong answers, challenge unsupported evidence, or correct the relevant error in practice.

Specify small changes that leave the correct answer unchanged and inspect how much explanations vary. Measure discovery and correction of errors, not just trust or preference. Include wrong-output explanations to determine whether they encourage appropriate doubt or persuade users to accept mistakes because the colors look credible. Stable pictures are not enough if the explanation directs attention away from the actual failure or omits important evidence.

Engineers check repeatability using identical versions and settings, with known-answer disturbances within the analysis scope. Do not tune explanations on test cases until they tell a preferred story. If explanations conflict with controlled changes, review baseline selection and assumptions. Grad-CAM and SHAP are model-investigation references, not evidence of complete transparency or certification of product correctness; the actual task still needs independent acceptance evaluation.

Limitations and Failure Cases

Coarse heatmaps can appear sensible even when predictions are wrong. Image perturbations can produce unnatural examples, and explanations do not replace acceptance tests. A colored region is a method-dependent summary, so reviewers should not read it as a complete record of all model dependencies or assume that a convincing visualization compensates for missing scale, timing, unit, or reference-answer evidence.

Perturbation changes data to explore its effect, requiring consideration of whether the example remains meaningful for the task. Replacing regions with flat color can introduce unfamiliar edges. Do not conclude that pale areas are irrelevant or dark areas are physical causes. Combine evidence sources and explicitly state questions the method cannot answer, rather than turning the display into an unsupported account of why an instrument or process behaved as observed.

Explanations add inspection possibilities but also computation and communication work. Misunderstood assumptions can create excessive confidence. Preserve acceptance tests and human-review paths separately from explanatory images. These are analysis options, not confirmation of existing product heatmaps or automatic explanations. Explanations must not replace checks of values, units, times, and downstream safety, because understanding a visualization does not establish that every part of the measurement and decision chain is correct.

Practical Checklist

• Are method, layer, and baseline specified?
• Are the limitations of the model explanation displayed?
• Are failure images included without encouraging overinterpretation?

References and Further Reading

• Selvaraju et al. — Grad-CAM
• Lundberg and Lee — SHAP