Data Ethics Club: The Alignment Community Is Unintentionally Building A Censor’s Toolkit

Article Summary

Artificial intelligence (AI) alignment is a direction of research aiming to orchestrate AI models so that they reflect human values. Alignment research also works on developing safeguards to prevent models from being susceptible to attacks from malicious actors such as through jailbreaks, prompt injections, and fine tuning. Unintended harms of alignment methods have been widely discussed in the field, such as failures to represent diverse populations in data leading to the systematic marginalisation of underrepresented groups. Whilst unintended harms have received research attention, the paper purports that intended negative consequences of alignment methods being leveraged by those with harmful intent has been neglected.

Alignment methods, the paper argues, are dual-use technologies weaponised for censorship and manipulation. Whoever defines the values that are “aligned” with fundamentally determines the goals the system serves. Techniques can be used for censorship as those controlling the system’s output can completely supress specific types of information. Alignment techniques can thus be leveraged to manipulate and distort user perception.

Possible malicious actors include state actors (governments, legislators, authoritarian regimes) and foundation model providers (large tech companies that train and own large models). These actors may leverage alignment methods for unethical intent in several ways: pre-training data filtering to remove unwanted content; post-training preference alignment to tune models to fit particular ideologies or preferences; and inference-time control, where hidden system prompts contain instructions defining the tool’s role, priorities, and constraints.

Several directions could be taken to reduce the likelihood of dual-use potential. Robust oversight mechanisms could be established to encourage transparency and verifiable evaluation, supporting public control and critique of models. Pluralism and competition should be cultivated in the LLM ecosystem to avoid concentrations of power and influence. So that users actively consider the potential for censorship and manipulation when they are selecting or using LLMs, efforts should work towards public awareness building.

“Alignment remains essential but must be pursued with clear recognition that it is a powerful tool that can serve both protection and oppression – depending on who wields it”.

Discussion Summary

We’ve seen dual use problems with social media censorship in the past, how is this problem with LLMs similar? How is it different?

We thought that the paper was interesting but unsurprising, as it seems clear that alignment approaches are taking political decisions, even if people don’t explicitly talk about them in this way. Alignment research focuses on how to adjust AI systems to reflect “values” which we hold as a society. Politics concerns making collective decisions about what is important to us: “politics is a battle of ideas, in which participants attempt to control the narrative through tapping deep-rooted values and beliefs. As decisions about which values to prioritise are decisions about what is important to us, they are political. Through this lens, it seems obvious that tools built for aligning AI with one set of values can be retrofitted for aligning AI for another, potentially contradictory, political purpose. There is little technical difference between censoring people making bombs from people talking about Palestine.

It is valuable for the dual use nature of alignment research to have been mapped out in this paper, yet some of us thought it potentially betrays some naivety on people working in this area. The paper argues that the alignment community need to be more conscious of the dual use nature of their work and that they need to be thinking about AI critically. What’s missing from this perspective is the fact that other disciplines have been talking about these topics for a long time. Conversations about social media censorship, such as the issue of LLM-based malicious bots, are nothing new. The problems being raised in the paper may arise from alignment researchers failing to engage properly with people in other subjects that have been voicing their concerns.

The realisation that alignment is not going to be suddenly “solved”, and that this research is not aiming for a stationary north star, is important. People working in AI are exposed to hype narratives and encouraged to believe that their work has a supernatural element (e.g. being told they are “unicorns” and “wizards”). These kinds of stories may make AI researchers more susceptible to thinking that they are the only ones able to solve certain problems, and more immune to recognising that the sorts of problems raised by this paper even exist.

Some of us have experience with trying to tackle the kinds of problems alignment research is addressing. For example, we’ve worked on projects trying to make tools for improving inclusivity in language use. Very quickly, we fell deep into the hole of trying to draw a line between “good” and “bad” language. There are lots of words which are widely used now but previously were disapproved of, and vice versa. Creating objective and generalisable distinctions is a highly complex and potentially infeasible task.

Underlying alignment research is a presumption that there is a ground truth which we are trying to align to. We don’t think this is true. The reason we ran into so many problems with inclusive language is that what we consider to be important or acceptable changes over time and between contexts. The goal of what we’re trying to “align” to will be different for different communities at different types.

As LLMs depend on a huge amount of data, and that data is littered with biases, it is really difficult to comprehensibly debias LLMs. For domains which we have less data about, such as under resourced languages, debiasing is even more difficult. Not only do we just have less data available to make accurate predictions about these areas, but those who design the systems largely don’t come from these cultural backgrounds and therefore don’t know how to evaluate them in ways which are culturally appropriate.

To some of us, alignment research feels like we’ve produced a knife and don’t want it to cut. Generative AI technology is simply so flexible, and people can apply it to so many things, that alignment seems to be a very uphill battle. In addition to the difficulties with identifying clear alignment goals, it might be the case that the actual technology we’re aiming at – large language models (LLMs) – isn’t well suited to being aligned. There might be alternative architectures which could embed human values better.

Given that there are deeply engrained and potentially unresolvable issues with the task, the goal of AI alignment perhaps should instead be to make people realise the reality of the nuanced world. Rather than deceiving people into thinking that nuance can be computed away, it should give people the tools to spot complexities.

One of the mechanisms the author stresses that will resist this problem is model pluralism (having lots of different LLM providers). Do you agree with this? Who has the power to make it happen?

Corporations are rushing headlong into the AI boom which is leading to real consequences felt by real people, yet, people aren’t being held to account for harm they cause. For example, OpenAI were not sued after they were responsible for cyberattacks on Hugging Face and others. We wondered why. Potentially, it is because OpenAI have a lot of power and resources available to them and their would-be complainants do not have the legal infrastructure to mount an effective challenge.

The OpenAI hacks did not happen because the technology overpowered constraints put in place by people; it happened because the people at OpenAI developed an unsecure environment. These are basic, human-centred mistakes. It seems unintuitive that people aren’t being held to account for them. Similarly, Grok has generated content which is outright illegal. It has been able to do this because of design decisions made by people. In any other context, people responsible for creating that kind of material would be in jail.

If the law was better equipped to handle AI, potentially these products would be pushed out much more slowly. There are not stringent enough laws in place to require companies preventing AI based harms from happening and there are not effective repercussions for those responsible for AI misuse. For example, perhaps deployers should not be allowed to put a model on the market until it has passed certain tests, as is the case in drug development. We wondered how the market would change if the legal consequences to these problems were meaningfully impactful, or even devastating, for the makers of these models. Currently, companies just offload consequences to “human-in-the-loop” or mythical “uncontrollable” AI.

As well as harms caused by AI being made by people, it’s important to remember that they are also felt by real people. AI Confidential with Hannah Fry takes a human perspective on AI risks, looking at stories emerging from real people and their use of AI. These people should be the forefront of decision making about how AI is governed.

The authors explicitly state that they don’t think that alignment research should stop. Do you agree/disagree?

The urgency of alignment dual use is vital. Large AI models are increasingly widespread and diffused across domains in our lives with an increasingly large potential for harm. As we have seen in referendums and elections across the world, AI used for propaganda and censorship can have global effects.

Responses to the arguments this paper is making need to include legitimate, and legal, repercussions for people that create problematic technology, even if it was originally created for a well-intentioned purpose. It is key that people consider the potential for dual use. This already exists in other domains like bioengineering. If people working in alignment research don’t want their work to be used for malicious intent, the onus is on them to prevent this from happening.

It is also true that a lot of or all technologies can be dual use and it is not always obvious how to decide when something should or shouldn’t be used. Asking communities whether they want to be served with an AI tool is one approach to identifying appropriate uses. However, this can raise the paradox of exposure: some communities are underserved by technology, meaning that certain tools don’t work as well for them as they do for others. Yet, by attempting to include them, those communities are at more risk of exploitation and surveillance. A history of having their data collected for reasons that are not to their benefit can make these communities reluctant to provide their data.

What change would you like to see on the basis of this piece? Who has the power to make that change?

There does seem to be political will for regulating AI, but the law hasn’t yet caught up. It is vital that the law addresses the current lack of transparency for the underlying value alignment of models that are deployed in society. Currently, very large online platforms (VLOPs) hold a huge amount of data about swathes of society. To ensure that this kind of power is used for the benefit of society and democracy, society needs to be able to look inside.

Transparency could be improved by adopting benchmarks for political or value alignment. It is reasonable for the law to require new models to have their political alignment to these benchmarks released. This would need careful consideration on how to define values, which needs careful consideration of who should have the power to make these decisions. It should be a democratic process, as it affects us all. To enable effective democracy, communication is essential; people need to be able to understand what they are deliberating about.

One way in which we can keep track of the rate of AI related incidents are through tools like the AI Incident Tracker, hosted by MIT, is a tool to classify AI incidents based on risks and harm severity. It tracks changes in incidents over time, the severity of harm associated with incidents, and more, to increase awareness and adoption of best practice AI risk management.

Attendees

• Huw Day, Researcher, University of Bristol: LinkedIn, BlueSky

• Jessica Woodgate, PhD Student, University of Bristol

• Vanessa Hanschke, Researcher, University of Bristol

• Paul Lee, investor and author

• Joyann Boyce, PhD Student, University of Bristol, LinkedIn

Data Ethics Club: I used AI. It Worked. I hated it.