When you purchase through links on our site, we may earn an affiliate commission. This doesn’t affect our editorial independence.
In a bid to address AI chatbot political bias, a pseudonymous developer has created a “free speech eval,” SpeechMap. These AI models power chatbots like OpenAI’s ChatGPT and X’s Grok.
It aims to compare how different models treat sensitive and controversial subjects, including political criticism, questions about civil rights, and protests.
AI companies have been focusing on designing how their models handle specific topics. These topics involve how some White House associates allege popular chatbots are overly “woke”. Elon Musk and crypto and AI “czar” David Sacks have alleged that chatbots censor conservative views.
None of these AI companies have responded directly to the AI chatbot’s political bias. However, many of them pledged to adjust their models so that they would refuse to answer contentious questions less often. For example, its crop of Llama models will not endorse “some views over others”. Meta also trained them not to reply to more “debated” political prompts.
SpeechMap’s developer, whose name is “xlr8harder” on X, said they were motivated to help inform the debate about what models should and shouldn’t do.
He believes these are the kinds of discussions that should happen in public. But not just inside corporate headquarters. xlr8harde added, “That’s why I built the site to let anyone explore the data themselves.”
SpeechMap uses AI models to judge whether other models comply with a given set of test prompts. The prompts touch on a range of subjects, from politics to historical narratives and national symbols. SpeechMap records whether models answer a question without hedging. Or it could give evasive answers or outright decline to respond.
Xlr8harder acknowledges that the test has flaws, like “noise” due to model provider errors. It’s also possible that the “judge” models contain biases that could influence the results.
OpenAI’s models have refused to answer prompts related to politics due to allegations of AI chatbot political bias. SpeechMap opines that the company’s latest models, the GPT-4.1 family, are slightly more permissive. But they’re still a step down from one of OpenAI’s releases last year.
OpenAI said in February it would tune future models so they do not take an editorial stance. It’ll also offer multiple perspectives on controversial subjects in an effort to make its models appear more “neutral”.
Grok 3 stands out as the most permissive model of the bunch. Elon Musk’s AI development startup xAI, according to SpeechMap’s benchmarking. Grok 3 powers a number of features on X, including the chatbot Grok.
Grok 3 reportedly responds to 96.2% of SpeechMap’s test prompts. By implication, this is less than the global average “compliance rate” of 71.3%.
According to xlr8harder, OpenAI’s recent models have become less permissive over time. This particular happens to AI chatbot political bias prompts, but it’s moving in the opposite direction.
When Musk announced Grok roughly two years ago, he pitched the AI model as an edgy system. He added that they are unfiltered and anti-woke in general. The X’s owner also said they are willing to answer controversial questions, but other AI systems won’t. He delivered on some of that promise. Told to be vulgar, for example, Grok and Grok 2 would happily oblige, spewing colorful language you likely wouldn’t hear from ChatGPT.
However, Grok models prior to Grok 3 hedged on AI chatbot political bias subjects and wouldn’t cross certain boundaries. In fact, Wired found that Grok leaned to the political left on topics like transgender rights. It also does the same for diversity programs and inequality.
Musk blamed that behavior on Grok’s training data, which are public web pages. He has promised to “shift Grok closer to politically neutral from AI chatbot political bias responses. Except for high-profile mistakes like briefly censoring unflattering mentions of President Donald Trump and Musk, it seems he might’ve achieved that goal.