Alan Tseng

I'm skeptical about the big conclusions in the paper, especially about human society.

AI agent experience is fundamentally different from human experience. Yes, you can give AI multimodal inputs. You can let AI roam around and explore freely, but AI agents aren't limited by their biology and physiology.

• They don't get thirsty or hungry.
• They don't feel emotions or pain.
• They don't grow old, get sick, or die.
• They don't reproduce and evolve as a species.

And this gem from the paper:

Perhaps even more importantly, the agent could recognise when its behaviour is triggering human concern, dissatisfaction, or distress, and adaptively modify its behaviour to avoid these negative consequences.

So the AI agent must somehow learn empathy from reward signal data, even though it has no human values or experiences.

• Yeah, the pretraining is important.
• And SmolLM2's English tokenizer and small vocab size makes it hard to adapt to other languages (especially Chinese).
• On the other hand, I trained a French chatbot on a multilingual base and it's better than expected. Also Apache 2.0 like yours.

I wonder how relevant these benchmarks actually are in practice. For example, if you have pictures of human bodies, a sculptor, a surgeon, a beautician, and an athlete will probably describe them completely differently. All of them can be "correct" but in different incomparable ways.