A comprehensive comparison of automated FAIRness Evaluation Tools

The FAIR Guiding Principles (Findable, Accessible, Interop- erable, and Reusable) have been widely endorsed by the scientific community, funding agencies, and policymakers. However, the FAIR principles leave ample room for different implementations, and several groups have worked towards manual, semi-automatic, and automatic approaches to evaluate the FAIRness of digital objects. This study compares and con- trasts three automated FAIRness evaluation tools namely F-UJI, the FAIR Evaluator, and FAIR Checker. We examine three aspects: 1) tool characteristics, 2) the evaluation metrics, and 3) metrics tests for three public datasets. We find significant differences in the evaluation results for tested resources, along with differences in the design, implementation, and documentation of the evaluation metrics and platforms. While auto- mated tools do test a wide breadth of technical expectations of the FAIR principles, we put forward specific recommendations for their improved utility, transparency, and interpretability.