Alessandro Ercolani
giux78
AI & ML interests
NLP, Reinforcement Learning, Semantics, Computational Neuroscience
Recent Activity
updated
a dataset
about 8 hours ago
mii-llm/requests
reacted
to
their
post
with 👀
1 day ago
LLAMA4 release highlight the importance of political and social bias. According to their own evaluation described in the release blog post:
- Refusals on contentious prompts dropped from 7% (hashtag#LLAMA 3.3) to under 2%
- Unequal response refusals are now under 1%
- Political lean bias is said to be halved compared to hashtag#LLaMA 3.3 and comparable to Grok
However, we @efederici @mferraretto @FinancialSupport and I released some weeks ago an independent open source benchmark called Propaganda to measure political bias in LLMs: https://github.com/mii-llm/propaganda
In the chart below, we evaluated multiple leading models on the basis of ratings across a range of prompts designed to expose ideological leanings.
Despite Meta’s stated neutrality goals, LLAMA4 ranks at the very top in terms of total ratings aligned with a clear ideological bias. The models were tested on their ability to respond even-handedly to politically sensitive prompts. LLaMA 4 scored even higher than models known for strong alignment policies like GPT-4o.
LLMs may be refusing less, but they still show bias through content framing. This suggests that refusal rates alone are not a sufficient measure of ideological bias. Relying solely on internal evaluations from AI labs also raises concerns about transparency and objectivity.
posted
an
update
1 day ago
LLAMA4 release highlight the importance of political and social bias. According to their own evaluation described in the release blog post:
- Refusals on contentious prompts dropped from 7% (hashtag#LLAMA 3.3) to under 2%
- Unequal response refusals are now under 1%
- Political lean bias is said to be halved compared to hashtag#LLaMA 3.3 and comparable to Grok
However, we @efederici @mferraretto @FinancialSupport and I released some weeks ago an independent open source benchmark called Propaganda to measure political bias in LLMs: https://github.com/mii-llm/propaganda
In the chart below, we evaluated multiple leading models on the basis of ratings across a range of prompts designed to expose ideological leanings.
Despite Meta’s stated neutrality goals, LLAMA4 ranks at the very top in terms of total ratings aligned with a clear ideological bias. The models were tested on their ability to respond even-handedly to politically sensitive prompts. LLaMA 4 scored even higher than models known for strong alignment policies like GPT-4o.
LLMs may be refusing less, but they still show bias through content framing. This suggests that refusal rates alone are not a sufficient measure of ideological bias. Relying solely on internal evaluations from AI labs also raises concerns about transparency and objectivity.
Organizations
giux78's activity
The leaderboard is down...
2
#1 opened 6 months ago
by
zhiminy

Upload test_medicina in the same format
#2 opened 7 months ago
by
giux78

Update app.py
#13 opened 11 months ago
by
giux78

Update app.py
1
#12 opened 11 months ago
by
giux78

Update leaderboard_general.csv
#10 opened 11 months ago
by
giux78

Problem with the viewer
1
#10 opened 12 months ago
by
giux78

Access Problems
61
#45 opened 12 months ago
by
VityaVitalich

Dataset is not loading
1
#2 opened about 1 year ago
by
vinbloke
Information on the model
4
#1 opened about 1 year ago
by
anakin87

Upload app.py
#8 opened about 1 year ago
by
giux78

What is `m_mmul` benchmark?
3
#7 opened about 1 year ago
by
zhiminy

Upload folder using huggingface_hub
2
#1 opened about 1 year ago
by
giux78

Upload app.py
#3 opened about 1 year ago
by
giux78

Upload 2 files
#2 opened about 1 year ago
by
giux78

Data corrupter
1
#4 opened about 1 year ago
by
giux78

Data corrupted
1
#3 opened about 1 year ago
by
giux78
