SoK: Measuring Compliance With Privacy and Data Protection Laws After the GDPR

As governments increasingly legislate around data practices, the research community plays a critical role in evaluating compliance and shaping accountability. But assessing legal compliance is inherently difficult: Law is nuanced, ambiguous, and carries consequences that can be reputational or monetary. In this paper, we systematize the prior academic work on measuring legal compliance with privacy and data protection laws, in the five years following the introduction of Europe's landmark privacy law-the GDPR-in 2018. We identify 84 computer science papers that measured legal compliance and analyze them according to how each paper approaches the technical, legal, and ethical aspects of their work. We find that nearly all studies found non-compliance (96%). Additionally, the vast majority of studies focus on the GDPR (77%) and, in particular, website-based analysis (43%). A small set of researchers note legal exceptions as possibly being applicable (26%), but even fewer investigate these exceptions. Less than half of papers detailed how they approached ethics (only 40% engaged with ethical considerations)-and those that did most frequently mention institutional ethics review as a general justification. Overall, disclosure of technical limitations (72%) was more common than legal (46%) or ethical (32%) limitations. Based upon our findings, we close with recommendations on how to improve research pursuing legal compliance measurement.