References
— Solon Barocas / Andrew D. Selbst, Big Data’s Disparate Impact, 104 California L. Rev. 671 (2016)
— Bryce Goodman/Setz Flaxman, European Union regulations on algorithmic decision-making and a ‘right to explanation”, research paper 2016
— Mireille Hildebrandt, Smart Technologies and the End(s) of Law, Edward Elgar 2015
— Joshua A. Kroll / Solon Barocas/Edward W. elten / Joel R. Reidenberg/ David G. Robinson / Harlan Yo, Accountable Algorithms, 165 U. Pa. L. Rev. 633 (2017)
— David Lehr/Paul Ohm, Playing with the Data: What Legal Scholars Should Learn About Machine Learning, 51 Univ. of California D. L. Rev. 653 (2017)
— Frank Pasquale, The Black Box Society, Harvard University Press 2015
15 Evaluation and regulation of algorithms: summary of discussions
Xavier Serra
Universitat Pompeu Fabra
The panel on Evaluation and regulation of algorithms included five presentations addressing different topics and addressing them from very different points of view. Alessandro Annoni, from the Joint Research Centre of the EU, talked about “Digital transformation and artificial intelligence: the policy-oriented perspective” where he explained the current and future policy EU initiatives related to AI from a regulation perspective. Martha Larson, from Radboud University and TU Delft, talked about “Reality, requirements, regulation: points of intersection with the machine learning pipeline” in which she emphasized the proper development of benchmarking for evaluating information retrieval systems (see Section 11). Anders Jonsson, from the UPF, talked about “Benchmarks and performance measures in artificial intelligence” in which he presented various approaches to benchmarking in AI from an academic perspective (see Section 12). Ansgar Koene, from the University of Nottingham, talked about “The IEEE P7003 Standard for Algorithmic Bias Considerations” in which he described the IEEE initiative on Ethics of Autonomous and Intelligent Systems and the different standards that are being developed under it (see Section 13). Finally, Heike Schweitzer, from Freie Universität Berlin, talked about “Algorithms and markets -a need for regulation??” in which she introduced a legal perspective by emphasizing the challenges that AI technologies bring to lawyers, presented in Section 15.
There was an initial list of questions that the organizers proposed for discussion: How should algorithms be evaluated in a research vs industrial context? Which are the policy needs in terms of the usage of algorithms into real applications? Which is the research needed to support these policy needs?. There were some relevant contributions to these questions during the talks, but there was no time for discussion to address them in depth. A much more extensive and focused discussion would be needed.