Data Ethics Club: Amazon’s Just Walk Out technology relies on hundreds of workers in India watching you shop#
What’s this?
This is summary of Wednesday 8th May’s Data Ethics Club discussion, where we spoke and wrote about the claims that Amazon’s Just Walk Out technology relies on hundreds of workers in India watching you shop. The article summary was written by Vanessa Hanschke and edited by Jessica Woodgate. The discussion summary was written by Jessica Woodgate , who tried to synthesise everyone’s contributions to this document and the discussion. “We” = “someone at Data Ethics Club”. Huw Day, Vanessa Hanschke, Amy Joint, Nina Di Cara and Natalie Thurlby helped with the final edit.
Article Summary#
Amazon’s Just Walk Out stores have been touted to provide a seamless shopping experience where customers can automatically pay by placing objects in their physical shopping basket. The contents are analysed using computer vision technology, thereby customers completely avoid interacting with a cashier. However, according to a report in The Information, an unnamed person claims that in 2022 as many as 700 out of 1000 sales had to be reviewed by Amazon’s team in India. In response, Dilip Kumar, the vice president of AWS Applications, writes that “the erroneous reports that Just Walk Out technology relies on human reviewers watching from afar is untrue. Most AI systems, including the underlying ML models behind these technologies, are continuously improved by annotating synthetic (AI generated) and real shopping data. Our associates are responsible for this labelling and annotation step”. This story has broken along with Amazon’s announcement of replacing Just Walk Out with Dash Carts (essentially a smart shopping cart with a display, scanner and scales).
Just Walk Out is currently implemented in 27 of 44 Amazon Fresh stores and some Whole Foods stores. Other startups are testing similar technology in Aldi.
If you fancy some extra reading, this article by The Verge goes into more detail about the enormous amount of human labour behind AI development.
Discussion Summary#
What do you think about Amazon selling human reviewers as computer vision?#
Passing off human reviewers as computer vision leans on and encourages misconceptions in the public sphere about the limits of computer vision and AI generally. The Information claimed that sometimes it can take hours for customers to receive their receipts, lending support to the suggestion that human assessment was required. This raises transparency issues by marketing tools to be much more capable than they actually are. Just Walk Out is not an isolated incident; there are many examples of how the power of “AI” is down to copious amounts of low paid human labour. In particular, there is a prevalence of companies hiring people to perform lengthy and tedious annotation work, often in low paid and insecure positions. “Behind even the most impressive AI system are people – huge numbers of people labelling data to train it and clarifying data when it gets confused… Only the companies that can afford to buy this data can compete, and those that get it are highly motivated to keep it secret. The result is that, with few exceptions, little is known about the information shaping these systems’ behaviour, and even less is known about the people doing the shaping… These AI jobs are… work that people want to automate, and often think is already automated, yet still requires a human stand-in.”. The underbelly of the industry consists of plugging the gaps in AI capabilities through cheap labour dispersed across the world. We would not, therefore, be surprised if the claims in the article are true.
The employment of data labourers raises concerns for worker ethics as well as privacy and consent implications for customers. For worker ethics, even though annotation is creating many jobs, those jobs are often low paid and insecure. Companies should be transparent about their employment of annotators, so we can ensure workers are being treated according to adequate labour laws. In India, for example, there aren’t currently any specific laws regulating AI and labour regulations are not always enforced. For customers, the right to make informed decisions is withdrawn when they are misled by deceptive marketing. A step towards transparency would be implementing regulation requiring companies to disclose the amount of human labour which goes into technology. This does not seem to be an unreasonable request and would pay dividends in improving trust and public awareness of the reality of AI capabilities.
Public education concerning AI is not aided by mainstream media; whilst this article has generated some noise around Amazon, it was perhaps lost in the general news cycle. Mainstream coverage of AI tends to overhype technology, maintaining that more detail is “deemed too complex for a mainstream audience”. This cultivates alarmism, and a dichotomy where AI research is either bad and uncontrollable or responsible, vague, and non-commital.
If Amazon’s response to the claim is true, would that change the way you think of the technology? Why or why not?#
Generally, our reaction to Amazon’s response is one of suspicion; it would not be the first instance of the company lying. There is precedence for companies AI washing, and exaggerating the abilities of their technology. “Move fast and break things” is still a prevalent attitude, as there is more money to be made by releasing the technology early and dealing with consequences as they arise. Making mistakes is less consequential if you are rich, summarised as “fines are nothing but a price tag for the wealthy”. Financial motivations thus lend support to the potential of big companies deploying underdeveloped technologies.
Regardless of whether or not Amazon’s claim (that humans play a minor role in the system) is true, we see various ethical issues with the collection and usage of data. Privacy and security issues arise such as anonymisation, length of data storage, and data transfer between jurisdictions. Amazon claim that “Just Walk Out technology… doesn’t use or collect any biometric information… simply links a customer with their payment instrument. When shoppers enter the store, the technology assigns them a temporary numeric code, which serves as the shopper’s unique digital signature for that shopping trip. The system preserves the code throughout the shopper’s time in the store. When they exit, the code disappears, and if they come back, they get a new code.” Whilst this informs us that biometric data isn’t collected, it doesn’t say anything about how the data which is collected is handled. A lot of information can be inferred from the products that people purchase, some of which is very personal such as finding out people are pregnant.
Other ethical issues arise with the development of the system, including the data used for training and the validation procedure. Amazon state that Just Walk Out was trained using synthetic data. Synthetic data aims to preserve statistical properties of the original dataset whilst preventing privacy attacks. However, it has been found that it is still possible to deanonymize synthetic data .
If stores implementing Just Walk Out had signage explicitly informing customers how data is being handled and how the system is operating (e.g. you are being watched, the data will be stored for this amount of time), or impact assessments were performed, we wondered whether our reaction would change. If we were properly informed about company practices, we were not sure if we would be more willing to support them (which is probably why companies are not transparent in the first place).
What do you think of the various types of automation in grocery stores? Are they convenient, desirable, or worse for the shopping experience? Do you make use of them?#
Automation stemming from AI hype can lead to technology being deployed before it is ready, resulting in more mistakes and less convenience. The fascination of flashy new technology sometimes detracts from simple and inexpensive, but boring, solutions. We wondered how reliable systems like Just Walk Out are, including how easy it is for people to dispute mistakes in bills. When contemplating a new technology we should clearly ask why we need it and what metric we will use to assess it. Involving humans and customer more heavily in the design and rollout of technology would be benefit the alignment of technological process. However, this can mean that pipelines take longer, potentially confining more ethical processes to smaller and more socially responsible companies.
In the context of grocery stores, whilst automation (e.g. self-checkout machines) can be more convenient for some people, it is not desirable for everyone. Having social contact in stores means a lot for some people; humans desire connection yet the drive for efficiency and automation seems to be increasing isolation. There should be a balance between efficiency and catering for everyone’s needs in an accessible way. In some stores, this has led to scaling back of self-checkout machines, or the introduction of “relaxed” checkout lanes specifically designed for people with special needs or who want to take their time and chat with the cashier. Other examples of improving accessibility include having a “quiet hour” where lights are dimmed and checkout noise is lowered. This can make shopping much easier for autistic people, for example, by fostering a calmer experience. The more we turn to automation in previously social settings, the harder it will be to meet the needs of different groups.
On the other hand, we did consider some use cases where automation can be beneficial. People who receive benefits or food bank support may prefer being automatically processed, as they would not have to disclose publically that they are claiming benefits. In these kinds of instances it could be privacy enhancing.
Bonus question: What change would you like to see on the basis of this piece? Who has the power to make that change?#
Primarily, we would like to see more transparency from companies developing and deploying AI. In pursuit of improved transparency, the Data Workers Inquiry at the Weizenbaum Institute is talking to workers involved in AI moderation and data labourers to improve understandings of their perspectives. Making workers visible is the first step, followed by pressure on politicians and companies.
Attendees#
Huw Day, Data Scientist, Jean Golding Institute, @disco_huw
Paul Matthews, Lecturer, UWE Bristol
Kamilla Wells, Citizen Developer, Australian Public Service, Brisbane
Euan Bennet, Lecturer, University of Glasgow, @DrEuanBennet
Melanie Stefan, Computational Neuro person, Medical School Berlin