Our NIST FpTVE

FpVTE 2012 (Fingerprint Vendor Technology Evaluation) is a benchmark evaluation in an environment similar to an operational large-scale AFIS. The evaluation datasets were from operational datasets of the USVISIT, FBI, and DHS systems. The evaluation dataset contained approximately 10 million subjects. The FpVTE is considered the world’s most prestigious evaluation.

The evaluation contains 3 classes (A, B, and C) to benchmark technologies in different application scenarios. Participants could choose to take part in Class A, Classes A+B, or Classes A+ B+C. There were 22 vendors from various countries to register for the evaluation; 18 took part in the actual tests. Of them, 2 took only Class A; 1 took Classes A+B; and 15 ran the full test (A+B+C).

 

FpVTE 2012 (Fingerprint Vendor Technology Evaluation) is a benchmark evaluation in an environment similar to an operational large-scale AFIS. The evaluation datasets were from operational datasets of the USVISIT, FBI, and DHS systems. The evaluation dataset contained approximately 10 million subjects. The FpVTE is considered the world’s most prestigious evaluation.  

The evaluation contains 3 classes (A, B, and C) to benchmark technologies in different application scenarios. Participants could choose to take part in Class A, Classes A+B, or Classes A+ B+C. There were 22 vendors from various countries to register for the evaluation; 18 took part in the actual tests. Of them, 2 took only Class A; 1 took Classes A+B; and 15 ran the full test (A+B+C).

The evaluation took almost 2 years to complete. The full report can be found at the following addresses:

http://www.nist.gov/itl/iad/ig/fpvte2012.cfm

http://dx.doi.org/10.6028/NIST.IR.8034   


The following tables are simplified forms of Tables 21, 22, and 23 in the NIST report (Section 10 Ranked Results, pages 92-94). It should be noted that:

  • Only results from top-ranked algorithms are listed.

  • Vendors are identified by letters (see report). The 4 vendors (highlighted in the following tables) that were recognized as the most accurate were:

    • D – 3M Cogent

    • I – NEC

    • Q – Morpho

    • V – AA Technology Ltd. 

  • Vendors may submit 2 algorithms for each class, denoted by the number following the assigned letter (i.e. V1, V2).

  • FNIR (False Negative-Identification Rate) is measured at FPIR (False Positive Identification Rate) = 0.1%.


From the results, we can conclude:

  • In each class, the top performers were one of the 4 vendors mentioned above. This indicates fingerprint algorithms from these 4 vendors are all very accurate and very stable.

  • For the algorithms from these 4 vendors, not one vendor can beat every other in every evaluation class.

  • In the 2-finger benchmark, algorithms from AA Technology Ltd were the best performing.

  • In the 10-finger benchmark, algorithms from NEC were the best performing.

  • The accuracy rate for the algorithms from these 4 vendors are very close. The differences may not be statistically significant.

  • The previously mentioned 4 vendors (NEC, 3M Cogent, Morpho, and AA Technology Ltd) demonstrated a consistent and significant advantage over that of the other vendors.