Designing Accountable Systems#
What’s this?
This is summary of Wednesday 17th May’s Data Ethics Club discussion, where we spoke and wrote about the paper Designing Accountable Systems where Severin Kacianka and Alexander Pretschner present a method to express and compare different definitions of accountability using Structural Causal Models. The summary was written by Jessica Woodgate, who tried to synthesise everyone’s contributions to this document and the discussion. “We” = “someone at Data Ethics Club”. Huw Day, Nina Di Cara and Natalie Thurlby helped with the final edit.
Discussion#
Before diving into the discussion, I thought it might be useful to go over a couple definitions of key terms, as there are a few overlapping concepts in accountability that can be easily confused. These definitions are inferred from the paper in question, and a paper by Theodore M. Lechterman.
Accountability – the obligation to explain and justify conduct, and susceptibility to sanctioning, which is held by an agent with decision-making power. An agent is thus accountable to some other agent for some state of affairs according to some normative standard. Agents may be accountable without being responsible or blameworthy.
Responsibility – how individuals can be connected to their actions, and the consequences of their actions, in ways that make it appropriate to praise or blame them. According to some perspectives, responsibility necessitates certain types of mental states. Being responsible for wrongdoing may generate a reason to hold an agent accountable, but responsibility is not a necessary condition for accountability.
Causality - the study of how things influence one another, assuming everything has a cause, and effect follows in a predictable and linear manner.
Q1 How does this paper fit into your existing understanding and beliefs about what accountability should be?#
When thinking about attributing accountability, there are two questions we should answer: firstly, is this an event for which a party could be held accountable? Secondly, if someone could be held accountable, what kinds of parties might we consider as liable? To help us answer the first question, we suggested the legal term Res ipsa loquitur, which is the principle that merely the fact that some accident occurs is sufficient to imply negligence. In other words, just because an accident happens and ignoring all other features of the situation, the fact that it has happened is enough to imply that someone, somewhere has been neglectful. This principle may help us with understanding accountability of “black box” algorithms which make decisions that are difficult to interpret. Even though we might not understand why a decision was made, we can still assign accountability, because the fact of a decision being made is sufficient.
To help answer the second question, regarding what kinds of parties we might consider as liable, we thought that accountability might solely lie with humans or organisations, or with technical agents. Difficulties might arise with holding AI agents accountable, as AI operates in ways somewhat independent to human understanding and control. When attempting to hold organisations accountable, we questioned how this could be operationalised when fault is not admitted. Do we need cooperation from both sides (those assigning accountability and those to be held accountable) for accountability to be meaningful? What about situations where accountability seems dispersed, and it is hard to attribute it to a single party? For example, if some software crashes while a developer is on holiday, is it the fault of the developer, those covering for the developer while they are away, or other parties involved in creating and distributing the software?
Q2 Do you agree that an understanding of causality is necessary for accountability? Are there instances where this isn’t the case?#
The paper argues that causality is necessary for accountability, but by itself causality might not be sufficient. We thought that causality may not be necessary for accountability. This could be because, for example, people in a position of authority might not directly cause an accident, however their position means that they represent various responsibilities and can therefore be held accountable. We also thought that the concept of causality helps us to distinguish accountability from association. Perhaps accountability is a spectrum between causality, and responsibility, with high causality and high responsibility entailing high accountability, and varying degrees of association between the two reflecting varying degrees of accountability. For example, one could imagine scenarios where a party has high causality but low responsibility, so is only held partially to account, e.g. a traffic sign pointing in the wrong direction leading to a motorist damaging some building works. The motorist had low responsibility attributed to the building works, but they did cause the accident.
Weaponizing causality#
The point was raised about how causality can be weaponised, for example with politicians having agendas like “we will fix the pavements!” Statements like these are easy to measure and people can be held accountable to them. This may make politicians seem more trustworthy, by promising to do things which can be clearly seen by their constituents but perhaps have low impact on their responsibility as figures of authority. They prove themselves to be accountable for their actions, but are the actions of enough importance for them to be worthy of praise?
Case study: Uber#
An example of the convoluted relationship between causality and accountability we discussed was the incident of the first fatal incident of an autonomous Uber vehicle. When discussing this case, the National Transportation Safety Board (NTSB) clearly attributed responsibility to the highest levels of the organisation. The NTSB argued that “safety starts at the top” and the collision was “the last link in a chain starting from an organisation that didn’t prioritise safety”. They asserted that Uber could have done more to prevent the fatality. On a technical level, NTSB found that it was badly written software which contributed to the crash. Specifically, the software repeatedly changed its classification of Elaine Herzberg (who was killed), never classifying her as a human. Because the classification kept changing, the system was not keeping trajectories of each object Herzberg was classified as. This meant that the system could not realise that, whilst it wasn’t sure of what Herzberg was, there was a continuous object there. By frequently changing the classification, without keeping a trajectory, the system did not have long enough to respond. It could not plan from “there’s a bike” to “I will break”. It just kept realising “there’s a bike”, then “there’s an unknown object”, then “there’s a bike”. Prior to Herzberg’s death, Uber also had restrictions on swerving and hard breaking which were put in place in response to findings related to the comfort of passengers. These restrictions have now been removed.
This case demonstrates how messy things can get; we must consider the automated system, the pedestrian crossing where they shouldn’t have, the distracted driver, the engineers who disabled or limited parts of the system, the company who set it all up, the government who didn’t put enough regulations in… The list goes on! We were curious about why this case is different from a driver in a normal car hitting a pedestrian. In the case of autonomous systems, considering the distributed agency and the problem of “many hands” as well as opaque chains of decision making, things become much more complicated.
Some of us argued that most issues come from the incentives set up by companies deploying autonomous systems. In the Uber case, engineers had changed the braking system during testing leading to a wider range of things being flagged as “don’t stop for this”. We wondered what the incentive system would have been which made the engineers widen the range of things the car won’t stop for. If the performance of the car in a set of tests impacts how the employees are positioned in the company (e.g. playing into their chance of promotion), this may affect the way in which they engineer systems, diverting more power to personal motivations.
In addition to how incentives may affect motivations behind engineering decisions, it is also important to consider the data systems are trained on. Training data must be fit for purpose, and if it is incomplete in important areas the repercussions could be immense. There will be outliers in any application, and designers of systems (or those creating incentives for designers) should be accountable for how the system handles such outliers.
Case study: ChatGPT#
One of the most impassioned issues surrounding ChatGPT has been its disposition to make false accusations about people. In these scenarios, so far, the onus has been on the individuals falsely accused to push back. There are a few notable examples of this behaviour. In Australia, a mayor has started a legal bid over false claims that he spent time in jail. In another case, ChatGPT falsely accused a professor of sexual misconduct, citing a Washington Post article which had ever been written, saying that the incident occurred while he was employed at an institution where he never worked. This is problematic for many reasons, including the false citation of a reputable resource, giving the statement a deceptive appearance of false reliability.
The deception of false reliability is so convincing that even qualified lawyers have been duped to use citations later proven to be fake. Will this lead to people on trial for defamation using the defence “I used ChatGPT to write that”, regardless of whether or not they actually did? Does/should citing ChatGPT absolve you of accountability? Another issue that law practitioners will have to adapt to is that in defamation cases, ascertaining how many people the false claims have reached is important to attributing fault. However, the opaqueness of the deployment of ChatGPT means that it is difficult to discern this information.
As with the Uber case, we should question if it should be the user who is accountable for the output of ChatGPT. Is there a difference here to self-driving car scenarios? Our intuition tells us that there is. Perhaps it is because of the different modes of systems the two applications generally fall into. How decisions made by automated driving are used will mostly fall into “fast thinking”, reactionary categories, where users do not have time to consider and reason about the answers provided by machines. This perhaps absolves users of some responsibility. Reacting to decisions made by systems like ChatGPT will more commonly fall into “slow thinking” categories, as users have more time to digest the information presented to them.
Case study: Facial recognition#
The third case study we considered was facial recognition. The ramifications of this for social justice are significant, considering the well known problem of bias in such systems. Again, as with defamation cases and ChatGPT, the onus is on the individual to prove that they have been incorrectly accused. This contradicts the right to be presumed innocent until proven guilty. There are numerous examples of faulty facial recognition occurring, and the ramifications are huge. For one person, their name wasn’t cleared for a year. This included spending 10 days in jail. Some people arrested by faulty facial recognition resort to agreeing to plea deals. Even when cases have been proven innocent by law, socially it can take much longer for their names to be cleared in the minds of members of their community.
When accountability is ascribed, clearing the names of the falsely accused, it is often to management of systems and sloppy investigative work by law enforcement, implying that the systems themselves are treated as objective. Is this the most appropriate way of assigning accountability?
Fixing accountability#
So, how do we go about fixing these issues? It seems important to consider contextual factors like the domain, the agents, and different relations of power and competing interests. We suggested that in deciding which kinds of cases to address, we should prioritise the most critical cases, such as those resulting in fatalities. When thinking about where we should deploy systems like ChatGPT, arguably the best use cases are those that are expensive to generate, but cheap to verify.
When we are deploying systems, similar to medications, a list of possible “side effects” could be displayed before use. The more knowledge users have about these systems, the more clearly we can attribute accountability. Therefore, to improve attributing accountability in the use of ChatGPT, mechanisms similar to cookie opt-ins could be useful (is this another justification for and application of systems like Data Hazards…?). People should actively give their agreement that outputs from ChatGPT are unreliable and should be verified. Giving agreement implies that people have been given reasonable warning about the risks with using such systems. But is this enough? We should be careful to ensure that warning labels do not stall as a way for owners of systems to offload accountability concerns. Bilateral approaches may seem more appealing, involving improving informed acceptance of risk on the user end before use, and building transparency into systems themselves, to help how decisions were made and where accountability should be assigned (to designers, deployers, or users) after use.
Attendees#
Natalie Zelenka, Data Scientist, University of Bristol, NatalieZelenka, @NatZelenka
Nina Di Cara, Research Associate, University of Bristol, ninadicara, @ninadicara
Huw Day, PhDoer, University of Bristol, @disco_huw
Euan Bennet, Lecturer, University of Glasgow, @DrEuanBennet
Ismael Kherroubi Garcia, Founder & CEO, Kairoi, LinkedIn
Lucy Bowles, Data Scientist at Brandwatch LinkedIn
Alessia Costa (Wellcome Connecting Science, Wellcome Genome Campus)
Ushnish Sengupta, Assistant Professor, Algoma University, https://www.linkedin.com/in/ushnish/
Daniel Collins, PhD Student, University of Bristol