GreaseVision: Rewriting the Rules of the Interface

Siddhartha Datta; Konrad Kollnig; Nigel Shadbolt

GreaseVision: Rewriting the Rules of the Interface

Siddhartha Datta
, Konrad Kollnig
, Nigel Shadbolt

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

67 Downloads (Pure)

Abstract

Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. We put forth GreaseVision, a collaborative human-in-the-loop learning framework that enables end-users to analyze their screenomes to annotate harms as well as render overlay interventions. We evaluate HITL intervention development with a set of completed tasks in a cognitive walkthrough, and test scalability with one-shot element removal and fine-tuning hate speech classification models. The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions. Our contribution also enables researchers to study the distribution of multi-modal harms and interventions at scale.

Original language	English
Title of host publication	DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop
Editors	Max Bartolo, Hannah Rose Kirk, Pedro Rodriguez, Katerina Margatina, Tristan Thrush, Robin Jia, Pontus Stenetorp, Adina Williams, Douwe Kiela
Publisher	Association for Computational Linguistics (ACL)
Pages	7-22
Number of pages	16
ISBN (Electronic)	9781955917940
Publication status	Published - 2022
Externally published	Yes
Event	1st Workshop on Dynamic Adversarial Data Collection, DADC 2022 - Seattle, United States Duration: 14 Jul 2022 → 14 Jul 2022 https://dadcworkshop.github.io/

Publication series

Series	Workshop on Dynamic Adversarial Data Collection. Proceedings

Workshop

Workshop	1st Workshop on Dynamic Adversarial Data Collection, DADC 2022
Country/Territory	United States
City	Seattle
Period	14/07/22 → 14/07/22
Internet address	https://dadcworkshop.github.io/

Access to Document

Full TextFinal published version, 11 MBLicence: Taverne

Cite this

Datta, S., Kollnig, K., & Shadbolt, N. (2022). GreaseVision: Rewriting the Rules of the Interface. In M. Bartolo, H. R. Kirk, P. Rodriguez, K. Margatina, T. Thrush, R. Jia, P. Stenetorp, A. Williams, & D. Kiela (Eds.), DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop (pp. 7-22). Association for Computational Linguistics (ACL).

Datta, Siddhartha ; Kollnig, Konrad ; Shadbolt, Nigel. / GreaseVision : Rewriting the Rules of the Interface. DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop. editor / Max Bartolo ; Hannah Rose Kirk ; Pedro Rodriguez ; Katerina Margatina ; Tristan Thrush ; Robin Jia ; Pontus Stenetorp ; Adina Williams ; Douwe Kiela. Association for Computational Linguistics (ACL), 2022. pp. 7-22 (Workshop on Dynamic Adversarial Data Collection. Proceedings).

@inproceedings{82c8b1c820f24cf5a69fde5d9d161b85,

title = "GreaseVision: Rewriting the Rules of the Interface",

abstract = "Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. We put forth GreaseVision, a collaborative human-in-the-loop learning framework that enables end-users to analyze their screenomes to annotate harms as well as render overlay interventions. We evaluate HITL intervention development with a set of completed tasks in a cognitive walkthrough, and test scalability with one-shot element removal and fine-tuning hate speech classification models. The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions. Our contribution also enables researchers to study the distribution of multi-modal harms and interventions at scale.",

author = "Siddhartha Datta and Konrad Kollnig and Nigel Shadbolt",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 1st Workshop on Dynamic Adversarial Data Collection, DADC 2022 ; Conference date: 14-07-2022 Through 14-07-2022",

year = "2022",

language = "English",

series = "Workshop on Dynamic Adversarial Data Collection. Proceedings",

publisher = "Association for Computational Linguistics (ACL)",

pages = "7--22",

editor = "Max Bartolo and Kirk, \{Hannah Rose\} and Pedro Rodriguez and Katerina Margatina and Tristan Thrush and Robin Jia and Pontus Stenetorp and Adina Williams and Douwe Kiela",

booktitle = "DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop",

address = "United States",

url = "https://dadcworkshop.github.io/",

}

Datta, S, Kollnig, K & Shadbolt, N 2022, GreaseVision: Rewriting the Rules of the Interface. in M Bartolo, HR Kirk, P Rodriguez, K Margatina, T Thrush, R Jia, P Stenetorp, A Williams & D Kiela (eds), DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop. Association for Computational Linguistics (ACL), Workshop on Dynamic Adversarial Data Collection. Proceedings, pp. 7-22, 1st Workshop on Dynamic Adversarial Data Collection, DADC 2022, Seattle, Washington, United States, 14/07/22.

GreaseVision: Rewriting the Rules of the Interface. / Datta, Siddhartha; Kollnig, Konrad; Shadbolt, Nigel.
DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop. ed. / Max Bartolo; Hannah Rose Kirk; Pedro Rodriguez; Katerina Margatina; Tristan Thrush; Robin Jia; Pontus Stenetorp; Adina Williams; Douwe Kiela. Association for Computational Linguistics (ACL), 2022. p. 7-22 (Workshop on Dynamic Adversarial Data Collection. Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

TY - GEN

T1 - GreaseVision

T2 - 1st Workshop on Dynamic Adversarial Data Collection, DADC 2022

AU - Datta, Siddhartha

AU - Kollnig, Konrad

AU - Shadbolt, Nigel

PY - 2022

Y1 - 2022

N2 - Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. We put forth GreaseVision, a collaborative human-in-the-loop learning framework that enables end-users to analyze their screenomes to annotate harms as well as render overlay interventions. We evaluate HITL intervention development with a set of completed tasks in a cognitive walkthrough, and test scalability with one-shot element removal and fine-tuning hate speech classification models. The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions. Our contribution also enables researchers to study the distribution of multi-modal harms and interventions at scale.

AB - Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. We put forth GreaseVision, a collaborative human-in-the-loop learning framework that enables end-users to analyze their screenomes to annotate harms as well as render overlay interventions. We evaluate HITL intervention development with a set of completed tasks in a cognitive walkthrough, and test scalability with one-shot element removal and fine-tuning hate speech classification models. The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions. Our contribution also enables researchers to study the distribution of multi-modal harms and interventions at scale.

UR - https://www.scopus.com/pages/publications/85137550873

M3 - Conference article in proceeding

T3 - Workshop on Dynamic Adversarial Data Collection. Proceedings

SP - 7

EP - 22

BT - DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop

A2 - Bartolo, Max

A2 - Kirk, Hannah Rose

A2 - Rodriguez, Pedro

A2 - Margatina, Katerina

A2 - Thrush, Tristan

A2 - Jia, Robin

A2 - Stenetorp, Pontus

A2 - Williams, Adina

A2 - Kiela, Douwe

PB - Association for Computational Linguistics (ACL)

Y2 - 14 July 2022 through 14 July 2022

ER -

Datta S, Kollnig K, Shadbolt N. GreaseVision: Rewriting the Rules of the Interface. In Bartolo M, Kirk HR, Rodriguez P, Margatina K, Thrush T, Jia R, Stenetorp P, Williams A, Kiela D, editors, DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2022. p. 7-22. (Workshop on Dynamic Adversarial Data Collection. Proceedings).

GreaseVision: Rewriting the Rules of the Interface

Abstract

Publication series

Workshop

Access to Document

Other files and links

Fingerprint

Cite this