Patient and public involvement to build trust in artificial intelligence: A framework, tools, and case studies#
Whatâs this?
This is summary of Wednesday 19th Octoberâs Data Ethics Club discussion, where we spoke Patient and public involvement to build trust in artificial intelligence: A framework, tools, and case studies, a paper written by Soumya Banerjee, Phil Alsop, Linda Jones and Rudolf N. Cardinal.
The summary was written by Huw Day, who tried to synthesise everyoneâs contributions to this document and the discussion. âWeâ = âsomeone at Data Ethics Clubâ. Nina Di Cara and Natalie Thurlby helped with the final edit.
Introduction#
Do you agree that it is important to increase public and patient trust in AI?#
Many of us said yes. Itâs easier to say that we should do this, but harder to implement it in practise.
People often donât trust AI (or automated systems in general) because they donât trust the people behind the scenes. If people trust organisations, they are more likely to trust plans to do things with their data. Terminology is an issue and we need to get understanding across to people.
Itâs quite hard to make trust out of thin air. It can sometimes come across as manipulative. Perhaps instead of trust we want more education and understanding.
Thereâs a clear conflict of interest between the data scientist being the same people explaining stuff to the patients. Good general start but thereâs a conflict of interest there that would need to be addressed in a more systematic setting. Itâs more democratic than most research. But they didnât bring anyone in until they had their research questions. They should have asked the patients questions, but potentially quite hard for them to vocalise. But individuals donât have the birds eye view that the data scientists do.
Just because something is trusted, doesnât mean itâs trustworthy. Maybe weâd want to talk less about trust and more about understanding.
We donât know how much trust there currently is, because how much do people think about what their data is used for?
Many of us donât trust big tech solutions now, but when did we. In the 1950s/1960s trust was asserted but not necessarily clear - trust was perhaps taken rather than given.
If their doctors trust a new tech and recommend it, patients are more likely to go for it. But for clinicians, theyâre less like to recommend a tool. Theyâre more likely to recommend something that is a supplementary tool than a replacement treatment to the norm.
Ethics committees considering drugs untested on pregnant women gave rise to real concerns. AI is sci-fi to many and so awakens peoplesâ doubts and worries. We need to get beyond that immediate negativity if trust is to succeed.
In Australia every time you have a procedure you have an opt-out from your results being included in data aggregation. In the UK most people probably just think of some big scandals involving massive costs and attempts to integrate data.
If you want participants involved with data that requires you to be in a trusted research environment, then itâll be very unlikely that youâll be able to share all the stages of research with the patients, e.g. data analysis.
The study is a bit of a role model for others - this sort of patient participation is exactly whatâs needed if we are going to implement AI stuff in healthcare.
When should we let automated systems make these decisions for us?#
As Kamilla put it âif youâre using an AI model for treating my kidney problem, I want to trust that model - I donât really care about the Terminator coming in.â
If you know that an unintelligible black box model performs better, but canât explain it to a patient that itâs better than a more explainable model, then whatâs the best approach?
Why not run both? White box and black box. Many feel the bottom line is that a human should always be deciding. If you have a doctor reviewing results, the human interaction adds an extra layer - the doctor might not trust the models and that would affect their interpretation. Can we build an external validation process for this?
Is applying these models actually useful? What do we gain by implementing them? It turns out - yes they are! Computer systems are better at reading x-rays/scans than humans. e.g. Blood testing machines feed results to a model that then lists recommended treatment (e.g. medicine dosages, or blood product suggestions).
There are specific quality assurance processes that can be applied to software-as-medical-device. e.g. FDA proposal for SAMD
The co-design model is great, but the software developers/data scientists have the responsibility to establish a review procedure, test the outputs to make sure everything is properly validated, but then continue to review periodically. Leaving a model/AI to run on its own without review can quickly go wrong and have big impacts
Kamilla is working on how to do this for social decision making (e.g. judges in court) #shamelessplug #ILPC2022.
Legal precedent supercedes the models, dictating acceptable thresholds. If the model outputs are past those thresholds, an exception needs to be flagged⊠but as always - a human gets to make the final decision!
What would you consider good evidence that research advisory groups are successful in their aims of more ethical and/or trustworthy research?#
The paper doesnât show a lot of evidence that itâs working, even if we liked the paper. But what would good evidence look like to us?
Could we adopt this program elsewhere? Is there any repeatability? But we still need to be nuanced on how to measure success? If a study is capable of being reproduced at a later stage, that might amount to good evidence of success. Confidence is never universal and isnât representative of the public.
You can ask patients to demonstrate understanding. âWe showed you this earlier, now weâre showing you this. What do you find easier to understand?â Maybe getting them to explain it to a different patient so you can measure outreach. Itâs quite intensive and only had 6 patients, so best case scenario itâs still just a drop in the ocean.
Do the Research Advisory Groups (RAG) only deliver confidence for the individuals who have participated in the process? Some of us felt this type of study can only really be a success if it increases confidence among the broader patient community, not just those directly involved.
Informed consent as a model perhaps for assessment as to whether patients agree for their data to be included in the processes.
Is there increased trust? A better patient experience?
Having patients living with conditions being able to feed experiences directly to researchers and healthcare providers is hugely useful - many patients find that they know more about doctors about their conditions (because living with it tells you far more than any other way to learn about something) - maybe the research groups of patients engaging with researchers should be the norm for everything.
Nina gave an example about trying to study children in care after they become adults - you want to be able to co-produce everything from the start, but you canât get funding to do that without being able to get in touch with people first. Chicken and egg funding problem!
Pharmaceutical trials sometimes have reporting systems in place to report problems with models for example.
Do we want approval or trust from patients? These are two slightly different things. Perhaps we need understanding, then we can understand why we donât sometimes get approval and what methods can we use that get approval.
As long as the patients get to decide for themselves (which seem to be what the authors were encouraging), thatâs the priority, with the long term goal to get rid of misconceptions about AI.
Is there anything that you think research advisory groups should not have a say in? Or anything that they donât have a say in that you think they should?#
Is it too early? Do we know the right questions to ask patients? Should the RAG patients be co-authors on the paper? As much as possible without compromising patient confidentiality?
Does it depend on the views of the patients - should more sceptical patients get more information to help convince them itâs trustworthy?
Thereâs a risk of overwhelming the patients with information. Are we going to give the patients their medicine plus homework?
To what extent do the patients have veto power? Canât they just ask another group of 5 people until they get approval? Itâs unclear how much their current say actually matter. In product development, the veto can be a setback to product development because things would be less likely to sell. Take what they donât like on board and go back.
The power dynamic comes into it a lot. Are you just asking the patient or are you asking them with a doctor present? How might they be influenced? Should you ask patients their views on various stakeholders?
We would need to make sure our selection of participants was as broad as possible, including considering cultural background, any cognitive impairments or mental health backgrounds. The paper did a good job of addressing this (e.g. with bipolar people). In all cases there seemed to be a high level of trust between doctor and patients.
How do we reach people who are distrusting of doctors? If youâre going to volunteer for this, then youâre probably not going to be someone who trusts your doctor.
Attendees#
Name, Role, Affiliation, Where to find you, Emoji to describe your day
Natalie Zelenka, Data Scientist, University of Bristol, NatalieZelenka, @NatZelenka
Nina Di Cara, Research Associate, University of Bristol, ninadicara, @ninadicara
Huw Day, PhDoer, University of Bristol, @disco_huw
Euan Bennet, Lecturer, University of Glasgow, @DrEuanBennet
Zoë Turner, Data Scientist, NOttinghamshire Healthcare NHS Foundation Trust @Letxuga007
Paul Lee, investor
Helen Sheehan, PhD Student, University of Bristol -Kamilla Wells, Citizen Developer, Brisbane