Quote: Are LLMs worth it?

Nicholas Carlini on 2025-11-19:

I briefly looked through the papers at this year's conference. About 80% of them are on making language models better. About 20% are on something adjacent to safety (if I'm really, really generous with how I count safety). If I'm not so generous, it's around 10%. I counted the year before in 2024. It's about the same breakdown.

And, in my mind, if you told me that in five years things had gone really poorly, it wouldn't be because we had too few people working on making language models better. It would be because we had too few people thinking about their risks. So I would really like it if, at next year's conference, there was a significantly higher fraction of papers working on something to do with risks, harms, safety--anything like that.

I agree with Nicholas Carlini — for folks thinking that LLMs are a valuable technology, it is imperative that they work on mitigating the increasingly clear consequential problems of the technology. For my part, as I'm not directly working on the models, it's my responsibility to go above and beyond on ensuring we do not collect unnecessary customer data, ensuring we protect data we have collected, and advocate fiercely against inappropriate LLM usage. This means no unconstrained chatbots, controlled model invocation to ensure value, and requiring evaluations and customer feedback to monitor quality.