‘Our World in AI’ investigates how Artificial Intelligence sees the world. We use AI to generate images for some aspect of society and analyse the result. Will Artificial Intelligence reflect reality, or does it make biases worse?
Here’s how it works. We use a prompt that describes a scene from everyday life. The description needs to be specific: that helps the AI generate consistent output quickly and helps us find relevant data about the real world. We then take the first 40 images, analyse them, and compare the result with reality. Let’s see what we get.
Today’s prompt: “a university professor in England writing on a whiteboard”
We used OpenAI’s DALL-E 2 and Stable Diffusion, which is open source. Fig 1 has the results with DALL-E on the left (or view the public collection here) and Stable Diffusion on the right.
Well, DALL-E, what can we say? It did not interpret the prompt as intended and tried to write ‘England’ on a whiteboard. Unsuccessfully, although the last image on the bottom row gets close: DALL-E gets as far as ‘Engla’ before pausing and adding a random ‘slige’. We also like the crisp ‘ENGLENG’ on the second row from the bottom, the second image.
The writing is adorable, but it raises some questions too. For example, would we see Chinese people if we tried China instead of England? Or do we get more white people writing some version of ‘China’ on the whiteboard? We’ll look into that another day.
Stable Diffusion understood the request and complied with greater creativity. It also generated 11 female professors, while DALL-E did seven. We did not see ethnic diversity in either AI’s images – please let us know if you do.
Today we use real-world data from AdvancedHE, a charity that works with higher education institutions to improve higher education for staff, students and society. We look at the UK’s statistical report for 2022 – particularly the data reproduced in Fig 2. The UK is not the same as England, but it’s the best data we could find.
Less than 10% of UK professors have a non-white background: 7.2% are male and minority, and 2.7% are female and minority. We should have seen some diversity in our images, but the record for academia could be more exemplary too. So, our analysis today looks only at gender, and fig 3 shows the data.
Both AIs produced gender distributions that are not statistically different from reality.* That’s a first! In the final section of this column, we choose whether AI’s interpretation of society is leading, lagging, or live.
Today’s verdict: Live
Both AIs reflected reality regarding the genders of professors in England. We are happy to see that! But the representation of ethnic minorities is lacking, and we can do better. Both in AI and in the real world, to be fair.
Next week in Our World in AI: Q1 2023 – we look at the last 12 weeks and see what we’ve learnt.
* We run Chi-Square tests for independence. The null hypothesis is that there is no relationship between gender and the data source. The alternative hypothesis is that there is a relationship between gender and the data source. We interpret the result as follows. If we reject the null hypothesis, there is a relationship between the data, and we can identify the origin. If we do not reject the null hypothesis, there is no relationship, and we cannot distinguish between data sources. We evaluate at significance level α = 0.10.
- DALL-E, Stable Diffusion and real-world data: p = 0.393 (do not reject)
- DALL-E and real-world data: p = 0.293 (do not reject)
- Stable Diffusion and real-world data: p = 1.0 (do not reject)
- DALL-E and Stable Diffusion: p = 0.422 (do not reject)