Generative AI’s output can reflect bias. Bias is skewed; biased output is prejudiced, whether positively or negatively, by the data training set.

This video describes how bias appears in GenAI:

GenAI relies on human-created texts, video, and audio obtained from various sources. A primary source is the web. Given that anybody can post anything on the web, GenAI has access to a lot of biased and false material. The biases in the data it uses are reflected in its output.

Outside of academia, biased data have been shown to reinforce structural inequities. For instance, when biases are hidden in algorithms used to decide who gets a loan, a job, a hospital bed, or even parole, or when biases shape social media feeds, ride-share allocations, and more.

Researcher Anjana Susarla refers to this problem as "algorithmic harm."

Bias in GenAI Image Generation

A side-by-side comparison of the original image input by Rona Wang and the image that GenAI generated. The original image features a young Asian woman with long straight hair, brown almond-shaped eyes with natural eyebrows, and a wide nose. The image generated by AI features a young Caucasian woman with long, wavy brown hair, blue eyes with sculpted eyebrows, blue deep-set eyes with purple eyeshadow and lashes, and a narrower nose. — Source: Business Insider

Image creation can also generate problematic output.

Image creation can also generate problematic output. Take, for instance, the story of MIT student Rona Wang, who tried to use such an app to create a professional-looking headshot.

The result? Among other things, the app lightened her skin, shaded her irises blue, reshaped her eyes, and otherwise changed her facial features toward a Caucasian average. A fancy term for the damage this kind of bias can do is representational harm. (The optional resources section at the end of this module contains videos addressing this and "algorithmic harms.")

In related news, another image app generated white men 97% of the time when asked to show images of people in positions of authority. This not only reflects the current norm (88% of CEOs, CFOs, and COOs at Fortune 500 companies are, indeed, both men and white); it also amplifies it in a kind of positive feedback loop, which does not bode well when considering that the ethical output would move the needle in the other direction entirely.

Reducing Representational Harm

GenAI companies are taking steps to limit offensive output by limiting offensive input or training material. One way to do this is to offer consumers advice on "strategic prompting" to help steer the app in a less offensive direction.

Another way is to have an underpaid assemblage of human workers identify and tag toxic content to ensure that the system is not trained on specific content, or if it has already been trained, it ignores the offending materials. Of course, continually exposing vulnerable workers to such content is another ethical concern.

Regardless, some users are intent on demonstrating the fallibility of GenAI safety guardrails or the steps companies have taken to protect us. They may do this for nefarious purposes or to test the systems to ensure they work without regurgitating prejudices. They have successfully created numerous workaround prompts that ‘jailbreak’ toxic material.

Offensive, dangerous, and unjust content pervades the data available for chatbot training because of the racism, sexism, hate, and so on that are pervasive in society. A fix for that problem is likely beyond the purview of GenAI creators, so GenAI users must remain vigilant for how bias can impact output.

In our own testing of GenAI image generation, we found biases in cartoon images generated of astronauts.

The images depicting a male showed him wearing a traditional space suit used in the profession. In contrast, those depicting a female showed a non-traditional space suit, similar to a costume worn in a sci-fi movie.

What Can We Do? Prompt Detail and Refinement

How we use and prompt the AI tools will affect our results.
The more detail we provide in the prompt, the more numerous the descriptions are, the better the output will be.
In addition, further refine the output by using additional details or by asking the AI tool to add, remove, or change details.