Data Ethics Club: I used AI. It Worked. I hated it.#
Article Summary#
Mike Taggart is an AI security expert who tests AI-enabled applications. Taggart’s role requires a level of comprehension about how applications implementing generative AI work. Even though his job means that he needs to understand how generative AI works, Taggart is critical of the societal and environmental harms caused by the technology and says that he wishes it had not been invented.
In the article, Taggart describes his experience of using generative AI (Claude Code) for a software development project. Before starting, he read up about “best practices” for using AI tools. He formed a reasonably well-defined idea of what he wanted the output to look like and then used generative AI to extend his ideas. His general approach was to proceed in a way that maximised the chances of keeping the model on-task and within expected parameters. Taggart ensured that the model’s output has a deterministic measure of success or failure at any given time, which helped the model to keep track of its own progress using external context. Each step of the build applied test-driven development, generating tests for functions that would define project features and implementing each in turn. He reviewed every line of code the model generated, finding that for initial drafts very little needed to change.
Applying generative AI made things considerably faster for Taggart than the project would have otherwise taken. The only direct changes he made to the codebase were for elegance. Core underlying logic was solid from the start, and the resulting product is secure - even more robust and implementing more features than Taggart would have built without using generative AI.
Whilst the process resulted in a reasonable output which achieved the stated objectives, Taggart found the experience miserable. Most of his time was spent reading proposed code changes and pressing accept, which he almost always did. He highlights a temptation to accept all changes in one go, but Taggart warns about the risk of this. Once the user stops scrutinizing the model’s output, the probability of things going wrong rapidly grows.
Even though he read each proposed change, Taggart found it challenging to claim that he knows the codebase deeply. Usually in a coding project he would be the author of a story composed of a structure of interlinked ideas and goals that he shares with users. When using generative AI, he became the audience rather than the author, reading to gain understanding after things had been built. This shift in paradigm means that safety largely comes down to the user’s discipline, as it is so easy to trust the model to do the right thing and stop systematic checking.
As long as generative AI coding tools exist, Taggart highlights that they will represent a quick and cheap answer to shortsighted organisations. He argues that no policy or procedure will prevent over-reliance on it. The tools require expertise to validate, but by using them it’s likely that peoples’ expertise will be diminished and growth stunted. New developers will find it challenging to become experts. Only continuous hard work and dedication leads to mastery, but with generative AI it is too easy to take shortcuts. The models constitute an addictive substance that alters cognition and take agency from us.
Discussion Summary#
Have you used AI coding before? How have you found it?#
Coding is an area where we have found generative AI to be beneficial. Especially in the early stages of coding, we have achieved more by using generative AI. Claude is a favourite for code projects because of its memory functionality, but some of us have licenses for a range of models to compare them and keep up to date with any changes.
AI can hold different roles for different people. Use cases go beyond code to translation, text summary, ideation, analysis, and even knitting patterns (although we don’t think it’s that great for this). Depending on its user-friendliness, we prefer it for some of these applications than more others. We’ve also found that generative AI can be useful to function as a neutral intermediary between people when reflecting or interpreting recorded conversations. Sometimes, AI is better at this than people. We wondered if it could be applied to code in a similar way, as a lens for reflection and interpretation whilst developing projects.
Some of us have previously learnt coding the hard way (before generative AI) and have found generative AI tools impressive but feel the author’s dilemma. We also see a lot of potential for things that we could apply these technologies to but wish that generative AI had never been invented. The world is worse off in many ways directly because of these tools, yet several aspects of our lives are also made easier from their existence. We felt cathartic and vindicated listening to an experienced programmer talk about their experience; it’s nice to read something that reflects the nuances of generative AI having lots of ethical issues but also being very useful for some things.
Whilst generative AI can be applied to a range of things and allow us to do things more cleanly than we would have otherwise, we’re not convinced that these tools save significant amounts of time compared to what we were doing before. Outputs from generative AI often need several fixes, sometimes to the point where it would be quicker to do it ourselves. How far and fast you can go using generative AI for coding varies depending on what stage of the project you’re at. It can also vary depending on whether your problem is a common one which AI is likely to have been trained on. We’ve found generative AI to be less good at newer aspects of code, such as new libraries for specific programming languages.
Were you surprised by this article? Did it match your experiences?#
We also agreed with Taggart that generative AI has taken the fun out of coding. Large language models (LLMs) are very literal and yoyo between being overly sycophantic to overly critical when asking for feedback, which can be annoying. Our role becomes a managing role where we do code review rather than implementation. Disassociation can quickly happen no matter how experienced someone is in coding. One way of mitigating this is to throw in continuous prompts, so that the user is continually engaging and acting on a level which is not too abstracted from the final product.
When using generative AI, dopamine comes from checking and taking the code further towards next steps. The temptation to approve all changes in one go can be addictive. We try to combine its use alongside our learning, but it is almost too easy to just go along with the outputs and revert to generative AI for any questions we need answering. It is so new that we don’t yet know how to moderate our use of it.
The ease of being able to approve all code without review makes it a strong incentive, which brings significant risks and is a very passive process. In the article, extra features were produced in AI outputs which Taggart didn’t request but did keep. Taggart was engaged and able to verify and review the code, but we wondered how much attention most people would pay to the detail of these features. General end users might not care too much about how things are made as long as they work, reducing consideration of the user experience as a disincentive for using generative AI. This highlights the weakness of the “human in the loop” approach to AI safety. Without detailed instruction and attention, results can be superficial or not work properly.
The addictiveness of generative AI quickly becomes dependence as, longer term, we see skill deterioration in areas where we have used generative AI too much. Only reviewing code will lead to us losing the ability to evaluate it, and the ubiquity of generative AI makes it more likely people won’t know how to code in the future. Students coming across a knotty problem are more likely to go straight to get the answer from generative AI, rather than trying to solve it themselves.
AI based on machine learning (ML), which includes generative AI, is a collection of things that have happened in the past. By becoming increasingly dependent on it, there is a risk that we get stuck in the past and move away from novel things we might have done if we weren’t using a past-based artefact. If necessity is the mother of invention, and generative AI removes necessity, it also takes away our drive to invent.
Historically, we have lost significantly powerful ways of knowing and being and we wondered if this is happening again now. Oral transmission, for example, is an incredibly deep form of knowledge which is much reduced today compared to earlier human history. There are examples of oral traditions having remarkably accurate narratives of events that happened over 7000 years ago.
How do you feel about other people using AI for coding? How has that changed over time?#
People can take issue with AI on principle in a similar way to how some cyclists don’t like electric bikes. However, like electric bikes, generative AI helps those who find certain tasks hard go further and gain more benefits than they would otherwise. By using generative AI, some of us have been able to create projects that we wouldn’t otherwise have attempted, both at work and at home.
Generative AI can be helpful for students by providing a more rapid avenue of support than seeking human supervision. Interacting with these tools is a way to remove barriers and friction which come with pursuing feedback from a person. Asking lots of questions and repeating things might be annoying for a person but is quite easy with generative AI. We have also found that some of the explanations provided by generative AI can be helpful to try and understand the process.
However, there is an important difference between speed and deep learning. A forklift can move all the weights in the gym, but that isn’t the point of going to the gym. You need to lift weights to build muscles; you need to work hard and take time to learn. Writing the code is a valuable process in itself. We don’t just care that it’s been written, we care how it was done.
Experience is needed to be able to tell where generative AI outputs work best and where they are drifting from the actual task at hand. Maintaining the expertise to be able to do this is hard, and people need to slow down to learn, which is hard. Jumping straight to the end removes friction, yet friction is essential for learning: we learn from mistakes because they stick in our memory. The current generation of students is missing out on the opportunities that earlier generations had to make mistakes and force their way through the friction.
The importance of process is encapsulated in a recent OpenAI controversy. OpenAI claimed to have “solved” a maths problem using generative AI. However, this received pushback. Two mathematicians had made significant advancements in the problem using tools adapted from OpenAI. Before the researchers were able to publish their work, OpenAI mounted a colossal effort to solve it. Worries were raised that this kind of work could make human mathematicians obsolete, and people criticised OpenAI for jumping straight to the solution without recognition of the work it takes to get there. The process of proving a theorem is valuable; having proof without understanding is much less valuable.
Given that the potential for using AI to solve problems that were previously significantly more difficult and costly, we wondered where our position is and whether our roles are devalued by the existence of these tools. Perhaps the suggestions and research taste for finding the questions we want to answer remains with people, and generative AI is the mechanism by which we pursue these directions.
As with previous innovations, humans have eventually found where they best fit. One day, we’ll find where generative AI best fits too. However, the extent to which these tools will be “revolutionary” is yet to be seen. Historically, a more reasonable view of a tool’s use emerges as the hype which comes with their introduction dies away. We can be pretty certain that generative AI will be disruptive to code as we continue to find our feet with it.
Yet we will not be able to find our feet if we don’t know when we’ve fallen over, and this can be harder with generative AI. The user needs to look carefully to spot errors and get insight to see that they’ve stumbled, but generative AI presents too smoothly to make hurdles obvious. Friction is needed to prevent mistakes and bad actors slipping through. To review code, people need to know how it works.
What change would you like to see on the basis of this piece? Who has the power to make that change?#
We have security concerns with generative AI, both in terms of what happens to data inputted into generative AI used over the web, and in terms of the possibility of hallucinations. There is no getting away from the fact that our data is being breached. Companies are loading customer data into LLMs to figure out marketing strategies. The rise of the citizen developer is now meteoric, and anyone who has access to generative AI can build out a web-app that is hosted by the AI provider. The provider gives the appearance of security, leading users to willingly hand over business or client data. No controversies have happened with this yet, but we expect to eventually see them as people build generative AI-owned apps that become part of critical systems with real significant consequences.
As controversial cases emerge from problems caused by generative AI, issues will be battled out in court, which will then force regulation. Use cases playing out in the courts will force boundaries onto how we use these tools. There is currently no equivalent to the EU AI Act in the UK, and more regulation is needed. Regulation could include, for example, specifications on how AI code should be audited and quality controlled. This is especially important in safety-critical and medical domains.
Until the law solidifies, we are still figuring out a path. It’s important to remember that AI, social media, and analogous technologies are choices that we decide to participate in. We have agency over these decisions and need to remember that we can change how it plays out.
As a society, the time in which we didn’t use generative AI has passed. For coding, we can think about it as another level of program language evolution, just like previous moves from hardware, to machine code, to high level languages like Python. Unfortunately, however, whilst the practice is widespread, we don’t yet know the process. A whole generation has suffered because of social media and will miss out as society figures out how to navigate AI. The next stage involves finding the correct processes we need to put in place in order to check generated code.
Given how widespread and interwoven with our lives LLMs have become in such a short space of time, there is a concern that we won’t find these processes and regulations until it is too late. The current approach to AI feels very short sighted. An economic shift is needed as the AI industry is too biased towards quick gains and neglects long term impacts. Entry level positions, such as for software engineers, are rapidly disappearing. Longer term, as those who have been trained before the generative AI boom start to retire, there will be a huge skills gap. What would have been the following generation of junior engineers is being replaced with generative AI. At the time when more senior people are needed, there won’t be any candidates that have gained the skills necessary to adopt the roles.
Attendees#
Huw Day, Data Scientist, University of Bristol: LinkedIn, BlueSky
Jessica Woodgate, PhD Student, University of Bristol
Nabeel Akram, Data Scientist and PGR in Air Quality and Particulate Matter Dispersion, University of Manchester
Paul Matthews, Lecturer in Data Science, UWE Bristol
Kamilla Wells, Citizen Developer / AI Product Manager, Brisbane
Zoë Turner, NHS