Content Level: Overview / Implementation considerations
Principles and Suitability
Latency measures time per request or event, throughput measures work per unit time, and availability measures usable service under a stated definition. Model frames per second do not equal correct readings per second. The p50 is the median and p95 the ninety-fifth percentile, not the maximum; each describes a different part of measured behavior.
Users need correct, fresh values when required. A continuously running program may produce unreadable images or delayed results, so distinguish service readiness from data readiness. Frame rate can count repeated or unreadable images. It must not stand in for usable measurements with complete units that genuinely support decisions, particularly when the output stream includes rejections or retained older observations.
Engineers define matching start and end points for timing, separating warm-up from normal operation. The tail contains slower cases that may matter for deadlines, requiring distributions rather than averages alone. Availability needs a denominator and a definition of usability, with stale and missing-data statuses. A responding port or webpage does not demonstrate operation of the entire reading path.
Scroll the diagram horizontally to inspect all details.
Illustrative Industrial Example
Design example: time a gauge task from image capture until a value with its unit reaches the recipient. Separate decoding, queueing, inference, and network time to find delays. This follows the result users actually need rather than reporting only the duration of a model operation performed on an image already available in memory.
Suppose inference is fast but the input queue has accumulated images, making delivered readings old. A small model speed improvement may not fix the cause; inspect queue waiting and actual image age. Batching can increase completed work but the first image waits for others. Higher throughput is therefore not automatically a shorter wait for every request, even if aggregate processing efficiency improves.
Engineers track one image identifier through all stages to associate timings with the returned result. Use simulated planned camera loads without disrupting real work. Inspect expired requests and tasks left running after timeout, defining cancellation or late-result discarding. Otherwise retries can create duplicate work and a load larger than the user intended, distorting both the experiment and the service's behavior.
Study and Trial Workflow
- 1 Define timing endpoints, warm-up, and concurrent load.
- 2 Collect distributions under normal and planned peak loads.
- 3 Test timeouts, reconnection, and backlogs, documenting equipment and versions.
Evaluation and Metric Definitions
Report p50, p95, and tails appropriate to the deadline alongside throughput, errors, and sample counts. Separate service uptime from the proportion of genuinely fresh, readable results. These are complementary observations, not interchangeable indicators: quick rejection messages or continued process availability do not establish that the measurement task delivered usable values promptly.
Specify image sizes, concurrency, numerical formats, equipment, and measurement duration to define the result's scope. Count all requests, including failures and timeouts, rather than only successful ones that can make timing appear overly favorable. Separate time to a fresh usable value from time to a status message. A fast unreadable response is not rapid delivery of the value the task requires.
Engineers examine per-stage time distributions with memory and heat during controlled sustained work. If p95 changes markedly between measurement periods, investigate queues and incidents instead of choosing the prettiest short interval. Cross-machine measurements need clock synchronization and discrepancy reporting. SRE guidance supports system observation; citing it does not establish an SLA or the availability of any particular product.
Limitations and Failure Cases
Batching may improve throughput but increase waiting. Averages conceal intermittent slow periods, and cross-machine timing requires control of clock discrepancies. Claims about performance must specify the path and operating conditions being measured, rather than imply that one favorable average establishes every request's timeliness, the freshness of its data, or the capacity of a larger deployment.
Concurrent tasks compete for resources, so time may not increase linearly. Multiplying one-camera results to predict many cameras requires testing assumptions, not a capacity guarantee from one number. Lower resolution may improve speed while erasing needles or units, requiring joint reading-quality evaluation. Do not remove difficult images from speed tests when they belong to the actual operating scope.
Operational metrics need policies for exceeding limits, such as stale-data notices or human review. Averages must not hide important slow results. Detailed observations aid diagnosis but add storage and confidentiality duties. Conclusions describe experiments under recorded conditions, not universal speed or cost savings, and a single observation must not be labeled certification of response time in every situation.
Practical Checklist
- Does the measured time include waiting in processing queues?
- Are resolution, batch size, and numerical precision stated?
- Does sustained testing reach a stable thermal state?