Mobile phone benchmarking is inherently like a beauty pageant: everyone knows there are filters, but they still look to see who’s filter looks more like the real thing. As a result, this time it was more straightforward. UL Solutions directly removed REDMAGIC 11 Pro and REDMAGIC 11 Pro+ from the 3DMark performance rankings. The reason was not insufficient performance, but “not meeting the benchmark standards.” The two phones are suspected of recognizing 3DMark app names and then applying more aggressive performance strategies, allowing the public version to score up to 24% higher.
How to catch it: Same test, same workload, just change the app name, the score will drop
UL stated that they used Google Play’s public 3DMark App and an internal version with the same test content but “renamed” for comparison. Theoretically, the workload between the two is the same, and the scores should be close; however, the public version of REDMAGIC 11 Pro scores up to 24% higher than the internal version, and the difference is too large to be explained by error. The conclusion is that the phone is not dynamically tuned based on workload behavior, but is doing “App name detection”; this kind of optimization for specific benchmarks is prohibited by their manufacturer’s rules.
Even more ruthless is a setting that looks like a performance gear: “Diablo mode.” What they say is that when 3DMark is detected, the phone will apply a higher performance configuration, and this behavior is enabled by default and cannot even be turned off when running 3DMark. The problem is that UL’s rule does not prohibit you from providing performance mode, but requires that “optional mode must be turned off by default and explicitly selected by the user.” If you turn it on by default and only turn it on when a specific app appears, it doesn’t sound like a “performance mode for players” but a “performance mode for the charts.”