NPU Performance And TOPS
TOPS stands for “tera operations per second,” a unit used to describe how many arithmetic operations an AI accelerator can perform in one second. An NPU (neural processing unit) is the hardware block designed to run neural-network workloads such as image enhancement, speech processing, and on-device inference. TOPS is often advertised as a single headline number, which makes it feel like a direct measure of “how fast AI will run.” That shortcut fails because real inference speed depends on more than raw multiply-accumulate throughput.
In practice, TOPS usually reflects a peak capability under a specific workload type, precision mode, and dataflow. Vendors may count operations differently, choose different model layers to measure, or assume ideal conditions that do not match your actual apps. For example, a camera feature that runs a small denoising network can finish quickly even on a lower-TOPS chip, while a larger segmentation model can bottleneck on memory movement rather than compute. I noticed this mismatch most clearly when comparing two devices that both listed similar TOPS but delivered different frame rates in the same camera mode—one device spent more time waiting for data.
What People Get Wrong
The most common mistake is treating TOPS as a universal speed rating. TOPS does not include the cost of moving tensors from memory to the NPU, scheduling work across CPU/GPU/NPU, or handling pre/post-processing steps that sit outside the NPU. Another frequent error is assuming that “higher TOPS” means “better accuracy,” even though accuracy depends on the model, quantization method, and training quality.
TOPS also depends on precision. Many marketing sheets quote INT8 TOPS, while some workloads run in FP16, INT16, or mixed precision. If a device advertises 20 TOPS for INT8 but your app runs a model that cannot be quantized cleanly, the NPU may fall back to other paths or run fewer layers at the advertised precision. The software stack matters too: the runtime (for example, Android NNAPI or a vendor’s inference engine), operator support, and graph partitioning determine what portion of a model actually lands on the NPU. When operator coverage is incomplete, the NPU may handle only part of the graph, and the rest runs on the CPU or GPU, which changes the speed outcome.
Memory bandwidth and cache behavior can dominate. Neural networks often reuse weights and activations, but the reuse pattern depends on layer shapes and batch size. A chip with strong peak TOPS can still underperform if the model’s tensors do not fit well in on-chip SRAM, forcing frequent DRAM reads. That is why two chips with similar TOPS can show different latency for the same model, especially for real-time camera pipelines where each frame has a tight budget.
Dependencies Behind The Number
To interpret TOPS, you need to map it to the full inference pipeline. The NPU sits inside a system-on-chip that includes memory controllers, DMA engines, CPU cores, GPU blocks, and power management. The NPU’s peak throughput is usually measured with a specific operator mix and a fixed tensor layout. Real models include convolutions, matrix multiplications, normalization, activation functions, and sometimes custom operators that may not map efficiently.
Quantization is another dependency. INT8 quantization can reduce compute and memory cost, but it requires calibration and careful handling of outliers. If a model uses dynamic ranges that do not quantize well, the runtime may insert extra steps or keep some layers at higher precision. That can reduce the effective TOPS used for the workload. I once saw a vendor demo where the “fast mode” required a particular model variant; the same feature on a different app version ran slower because the model graph changed.
Thermals and power limits also affect sustained performance. Peak TOPS ratings typically assume a short burst at a target power level. In continuous workloads like video processing, the system may throttle clocks, and the NPU may reduce frequency to stay within thermal design power. That means a device can look strong in a benchmark that runs for a few seconds, then fall behind in longer tests.
How To Judge Real NPU Speed
Check Precision And Workload
Start by reading what the TOPS number refers to: INT8, FP16, or another precision mode. If the spec sheet does not state precision and the assumed model type, treat the number as a rough marketing indicator. Look for additional details such as “TOPS for INT8” or “peak TOPS under X framework.” When you compare two devices, compare the same precision category and the same class of workload (for example, vision inference versus speech). If a device lists only one TOPS figure without context, you can still compare it, but you should expect mismatches in real app behavior.
Use Latency Benchmarks, Not Peaks
Prefer benchmarks that report end-to-end latency for a known model or a known feature. For phones, a practical approach is to test the same on-device feature across devices and measure time-to-result under similar conditions. For example, measure how long it takes to run a face detection pass on a fixed-resolution image, then repeat with the same resolution and lighting. If you use a synthetic benchmark, pick one that reports per-layer or per-operator timing, because operator mapping can explain why TOPS does not translate to speed.
When you see a benchmark that runs for a short time, watch for warm-up and throttling. A mild frustration is that many benchmark pages omit whether the test includes thermal stabilization; the results can swing after the first few runs. A practical target is to run each test long enough to reach steady behavior, then report median latency across multiple trials.
Inspect Model Support And Fallback
Real performance depends on how the runtime partitions the model between NPU, GPU, and CPU. On Android, NNAPI can route operations to different accelerators depending on driver support and operator coverage. If an operator is unsupported, the runtime may fall back to CPU, which increases latency and reduces the effective use of the NPU. A useful method is to check logs or profiling output from the inference framework to see which operators execute on the NPU. Some developer tools expose operator placement; even without deep profiling, you can infer fallback when latency jumps for certain model variants.
Compare Sustained Tests Under Load
For camera and video features, sustained throughput matters more than peak TOPS. Run a test that repeats the same inference in a loop for long enough to trigger thermal behavior, then compare average frames per second or average per-frame latency. A simple measurement setup can use a fixed video clip and a consistent output resolution, then record processing time with a high-resolution timer. If one device maintains stable latency while another degrades after a few seconds, the first device likely has better sustained power management for that workload.
As a side observation, I often see devices that look close in short benchmarks separate in longer runs because the NPU frequency governor reacts differently to the same thermal envelope. That is where “TOPS” stops being a useful predictor.
Educational Case Examples
Scenario 1: Two phones with similar TOPS A buyer compares two phones advertising 15–20 TOPS for INT8. In a controlled test, both phones run the same on-device image enhancement feature on 12-megapixel stills. The higher-TOPS phone finishes in 120 ms for the first run, but its latency rises to 180 ms after several repeats, while the lower-TOPS phone stays near 140–150 ms. The difference comes from sustained power limits and memory traffic patterns in the enhancement model, not from the peak arithmetic rate.
Scenario 2: Speech model with mixed precision Another buyer looks at a laptop listing high TOPS for INT8. A speech-to-text app uses a model that the runtime executes partly in FP16 due to operator constraints. Profiling shows only a portion of the graph runs on the NPU, with the rest on the CPU. The device with lower advertised TOPS completes transcription faster because it has better operator coverage for the specific model graph and a more efficient partitioning strategy.
TOPS Vs Real-World Checklist
| What You See | What It Usually Means | What It Does Not Tell You | How To Verify |
|---|---|---|---|
| Higher TOPS | Peak arithmetic throughput under stated precision | End-to-end latency for your app’s model | Run the same feature test multiple times and compare median latency |
| TOPS for INT8 | Compute and memory assumptions for INT8 graphs | Performance when the model runs in FP16 or mixed precision | Check whether the runtime uses INT8 for that model; look for operator/precision notes |
| Benchmark score | A measured outcome for a chosen test model | Your specific app graph, resolution, and input pipeline | Prefer tests that match your workload class (vision, speech) and resolution |
| Sustained results | Latency or throughput after warm-up and thermal stabilization | Peak burst behavior | Compare average latency over a longer run (for example, tens of seconds) |
Step-by-step checklist for a purchase decision: pick the device class you care about, identify the precision category behind the TOPS claim, choose a benchmark or feature test that matches your workload, run repeated measurements to capture variance, then compare sustained behavior rather than only the first run. If you cannot find matching tests, treat TOPS as a tie-breaker at best, not a deciding factor.
Common Mistakes To Avoid
People often compare TOPS across different precision modes without noticing. A device advertising “TOPS” for INT8 cannot be directly compared to another device advertising “TOPS” for FP16 unless the spec clarifies the basis. Another mistake is trusting a single benchmark score that uses a model not representative of the apps you run. A third mistake is ignoring software maturity: an NPU driver update or runtime update can change operator mapping and performance, and older benchmark results may not reflect current behavior.
Some spec sheets also omit whether the number reflects peak theoretical throughput or a measured benchmark. When the documentation lacks a test model description, you cannot infer how much of the graph maps to the NPU. I have seen cases where a vendor demo uses a custom model graph that fits the accelerator well, while a third-party app uses a different graph that triggers fallback. That gap explains why two devices with similar TOPS can behave differently in real apps.
Finally, avoid using TOPS to predict battery impact. Power draw depends on the whole system: CPU wakeups, memory traffic, and display pipeline. A higher TOPS chip can still consume less energy for a given task if it finishes sooner, or it can consume more if it triggers higher power states. Without measured power data for the same workload, battery conclusions stay speculative.
FAQ
Does Higher TOPS Always Mean Faster AI?
No. TOPS reflects peak arithmetic throughput under specific assumptions, while real inference latency depends on model size, precision, operator support, memory bandwidth, and thermal throttling.
What Precision Should I Compare Between Devices?
Compare the same precision category (for example, INT8-to-INT8) and check whether the vendor states the precision basis. If precision is unclear, treat TOPS as a rough indicator and rely on latency tests.
Why Do Two Chips With Similar TOPS Perform Differently?
Operator mapping and fallback change how much of the model runs on the NPU. Memory movement and tensor layouts can also dominate runtime, so peak compute does not guarantee lower latency.
Are Benchmarks With One Number Reliable?
They can be useful if they specify the model, input resolution, precision, and whether results include warm-up and sustained conditions. Without those details, the score often cannot predict your app behavior.
How Can I Test NPU Performance Without Developer Tools?
Use a repeatable on-device feature test: run the same action multiple times on the same resolution, record time-to-result, and compare median latency across devices. If possible, repeat after several minutes to observe thermal effects.
Author's Insight
TOPS is a hardware throughput unit, not a direct latency guarantee. The gap between advertised TOPS and observed speed comes from software partitioning, operator coverage, precision constraints, and memory bandwidth limits. Evidence from common inference behavior shows that end-to-end performance often tracks the slowest stage in the pipeline rather than peak compute. A practical approach is to treat TOPS as context, then validate with workload-matched latency tests that include repeated runs and sustained conditions. If you see a runtime version number in release notes (for example, an NNAPI driver update dated 2024-xx), performance can shift even when the chip stays the same.
Key Takeaways
- TOPS describes peak arithmetic throughput under stated precision and workload assumptions, not your app’s end-to-end speed.
- Real performance depends on precision mode, operator mapping, fallback to CPU/GPU, memory movement, and thermal throttling.
- Compare devices using latency or throughput tests that match your workload class and include repeated runs for steady behavior.
- Use TOPS as a tie-breaker only when precision and benchmark context are clearly stated.
