QR Code is widely used in our everyday life. As a two-dimensional barcode, it can store more data than 1D barcodes.There are many open-source and commercial libraries or SDKs which can read QR codes. A benchmark is needed to evaluate which one is more robust or suitable for a specific use case. To know more go through this PDF or visit blog https://www.dynamsoft.com/codepool/qr-code-reading-benchmark-and-comparison.html

1. Best QR Code Reading
SDKs

2. What is QR code?
§ QR Code is a type of two-dimensional barcode
§ Consists of black squares arranged in a square grid on a white background
§ Stores more data than a 1D barcodes
§ QR Code has full (360 degrees) reading features
§ QR code has a finder pattern located in three of its corners

3. QR Code Reading Benchmark and Comparison
In the following slides we are going to run a benchmark based on an image data set to find out
the best QR Code reading SDKs.

4. Dataset
A comprehensive benchmark was made by the author of boofcv to compare boofcv with other
4 libraries
§ Performance test is made on the dataset to evaluate 5 open-source libraries
§ Dataset contains 536 images containing 1232 QR codes and has 16 categories
§ blurred
§ bright_spots
§ brightness
§ close
§ curved
§ damaged
§ glare
§ high_version
§ lots
§ monitor
§ nominal
§ noncompliane
§ pathological
§ perspective
§ rotations
§ shadows

5. Performance test tool
In order to run the benchmark, a performance test tool is written
§ The tool uses Python to run the test and provides a web interface. The Flask web
framework is used.
§ Code can be accessed here
§ It is going to run on a PC device with an Intel i5-10400 CPU and 16GB memory.

6. Evaluated libraries
• Dynamsoft Barcode Reader (DBR, version: 8.6, commercial)
• Commercial SDK A (version: Java 13.5, commercial)
• Commercial SDK B (version: Windows 5.19.3.10, commercial)
• BoofCV (version: pyboof 0.36.1, open-source)
• Zxing (version: 3.4.1, open-source)
• Zbar (version: pyzbar 0.1.8, open-source)
• WeChat QR code detector in OpenCV (version: OpenCV 4.5.3, open-source)

7. Evaluation metrics
Performance is evaluated by reading rate and runtime.
Reading rate can be achieved by dividing detected QR codes by total QR codes in images
If the value of IoU (Intersection over Union) between the detected QR code’s polygon and the
ground truth’s is over 0, then the QR code is considered detected.
Reading rate = Detected QR codes / Total QR codes
*More details about evaluation metrics can be found here

8. Detection results
Reading rate in percent
Categories Dynamsoft
Commercial
SDK A
Commercial
SDK B
Zxing Zbar BoofCV
OpenCV
Wechat
blurred 69.23 38.46 26.15 23.08 38.46 40.00 55.38
bright_spots 38.14 30.93 6.19 21.65 19.59 23.71 30.93
brightness 81.18 52.94 32.94 54.12 51.76 78.82 68.24
close 92.50 25.00 35.00 5.00 12.50 77.50 65.00
curved 73.33 38.33 33.33 30.00 35.00 50.00 43.33
damaged 53.49 27.91 23.26 20.93 25.58 16.28 48.84
glare 86.79 20.75 22.64 20.75 35.85 32.08 64.15
high_version 97.30 35.14 59.46 5.41 27.03 40.54 18.92

9. Categories Dynamsoft
Commercial
SDK A
Commercial
SDK B
Zxing Zbar BoofCV
OpenCV
Wechat
lots 60.48 97.86 1.90 82.86 18.10 99.76 0.00
monitor 100 5.88 5.88 0.00 0.00 64.71 94.12
nominal 91.03 62.82 51.28 51.28 66.67 89.74 80.77
noncompliant 88.46 3.85 15.38 19.23 50.00 3.85 92.31
pathological 100 78.26 78.26 34.78 65.22 43.48 91.30
perspective 68.57 34.29 51.43 37.14 42.86 82.86 42.86
rotations 99.25 67.67 63.91 42.86 49.62 96.99 82.71
shadows 100 95.00 80.00 65.00 90.00 85.00 85.00
Total average 81.23 44.69 36.69 32.13 39.27 57.83 60.24
Detection results
Reading rate in percent

10. Detection results
Dynamsoft Barcode Reader ranks 1st in most of the categories. It ranks 2nd in the noncompliant and
perspective categories and 4th in the lots category.
Reading rate in percent

11. Detection results
Runtime per image (in milliseconds):
Runtime Results
Engine Result
DBR 141.10
Commercial SDK A 314.04
Commercial SDK B 48.31
Zxing 181.68
Zbar 199.26
BoofCV 205.48
OpenCV Wechat 175.06

12. Detection results
Runtime per image by categories (in milliseconds):
Categories Dynamsoft
Commercial
SDK A
Commercial
SDK B
Zxing Zbar BoofCV
OpenCV
Wechat
blurred 298.00 335.16 39.67 206.89 177.58 138.16 125.31
bright_spots 252.84 625.47 47.56 311.59 319.81 221.81 112.25
brightness 192.75 600.14 47.86 320.32 349.79 273.43 134.79
close 169.40 607.27 80.08 295.18 381.65 150.70 676.00
curved 188.76 300.82 43.40 173.44 220.28 103.42 129.54
damaged 200.03 205.49 42.05 84.41 96.76 46.14 138.35
glare 104.66 217.60 45.40 130.32 157.30 70.80 95.38
high_version 106.06 349.21 79.88 294.97 141.70 268.36 566.03

13. Categories Dynamsoft
Commercial
SDK A
Commercial
SDK B
Zxing Zbar BoofCV
OpenCV
Wechat
lots 275.43 851.14 110.14 1073.00 1477.57 6149.00 139.57
monitor 176.59 812.53 105.88 233.29 515.47 142.47 422.76
nominal 74.05 203.65 32.12 141.49 111.68 109.49 54.54
noncompliant 58.88 81.63 34.44 47.81 57.06 31.94 43.31
pathological 8.52 4.09 65.87 3.00 1.83 14.61 19.39
perspective 25.74 46.34 14.91 27.66 24.31 26.74 24.60
rotations 72.11 173.84 35.80 106.84 134.66 178.45 86.91
shadows 102.21 243.50 42.29 150.00 163.29 117.50 86.36
total average 144.13 353.62 54.21 225.01 270.67 502.69 178.44
Detection results
Runtime per image by categories (in milliseconds):

14. Detection results
Although Commercial SDK B has low reading rate on the dataset, it is the fastest. The Dynamsoft
Barcode Reader ranks second on average.

15. Update Runtime Settings to Improve the Results
§ The benchmark is done without updating the SDKs’ default settings, which can
be updated to improve the reading accuracy or speed
§ Dynamsoft Barcode Reader provides rich parameters that users can customize
and optimize for different usage scenarios for the best scanning performance
§ The runtime settings of DBR can be represented and modified using a JSON
template
*Learn more about parameters of DBR here

16. Modify the Template to Decode the Lots Category
§ Reason - QR codes are relatively small and the
image has a large resolution (3024x4032) while in
DBR’s template, there is
a ScaleDownThreshold parameter which is set
to 2300 so that the image will be scaled down for a
better decoding speed.
§ If we set the ScaleDownThreshold to 99999, we can
decode all the QR codes
§ The reading rate for the lots category can increase
from 60.48 to 100 using the modified template
There are several files in the lots category that DBR cannot detect all the QR codes

17. Modify the Template to Decode the Blurred Category
§ For blurred QR codes as below, GRAY_SMOOTH image processing method can be used before deblurring
to decode it
§ If the QR codes are blurred too much, not much can be done
§ DBR can only decode one extra image after using the modified template in the blurred category. However,
SDKs like the Dynamsoft Camera Enhancer can help filter the blurred image
DBR comes with many image processing algorithms

18. Conclusion
§ The Dynamsoft Barcode Reader has the best QR code reading accuracy on the test set
§ DBR speed is good and has rich runtime settings to adapt to different scenarios
§ If conditions allow, it is better to choose DBR as the barcode reading SDK

19. Links
Download Offline SDK
Get 30-day FREE Trial License
Links related to Dynamsoft Barcode Reader