![[ Center for Humane Technology ]](https://substackcdn.com/image/fetch/$s_!uhgK!,w_120,h_120,c_fill,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5f9f5ef8-865a-4eb3-b23e-c8dfdc8401d2_518x518.png)
How do AI systems affect humans? How do they impact our well-being, the way we think, our relationships with one another — and our communities?
Those are questions we’ve all been asking ourselves more and more as AI becomes entangled with our daily moments. But it’s difficult to find empirical answers, and that’s a problem.
Despite the plethora of technical evaluations and benchmarks which test AI competence on tasks ranging from computer hacking to image recognition, there is frustratingly little research on those much more urgent and important questions. Which is why the Center for Humane Technology has launched our new program: Humane Evals.
We first introduced this work back in April. We argue that the harms associated with AI – everything from tragic teen suicides, to subtle but widespread ‘cognitive offloading’ – show the need for more measurement, understanding, and communication of AI’s psychosocial impacts. We also believe that what’s needed isn’t a single conclusive benchmark or analysis but rather a growing interdisciplinary field of research and practice focused on these questions.
Since then, we’ve begun that work in earnest — and we’d like to share what it’s starting to look like:
Telling the human stories behind Humane Evals
We’ve just released the first of a new series of Humane Evals episodes on our podcast Your Undivided Attention. In it, Aza and I sit down with researcher Jared Moore, who’s been at the forefront of studying the psychological harms of AI chatbots. We discuss the measurement gap that Humane Evals is trying to fill, why this work is so critical and timely, and what we hope to accomplish. Along with posts on this Substack, we’ll use our platforms to highlight the urgent questions and innovative work that are driving this new field forward.
Introducing our own prototype to evaluate AI anthropomorphism
This fall, we’ll publish our proof-of-concept evaluation of anthropomorphic behavior in consumer-facing LLMs. As well as showing the extent to which different AI systems pretend to have human characteristics like emotions and desires, we’ll also show how current methods can only do so much — and why we need better techniques to see how AIs behave in the real world.
Working to solve the missing research data problem
Right now, many attempts to study the psychosocial impacts of AI are stymied by a single, shared problem – a shortage of good data. We think we can identify a solution to this, but if you’re a researcher, we need your help defining the essential characteristics of good datasets. Get in touch for a conversation with our collaborator on this project, Meredith Wade.
Connecting expertise from across domains
This work is interdisciplinary – it needs insights from psychology, machine learning, tech policy, human-computer interaction, digital entrepreneurship, social science, and much more. People in these fields don’t always encounter each other’s work or know that they’re experiencing the same roadblocks, so we’re piloting a new series of small, invite-only events to foster new connections and establish shared agendas. If you’d like to be considered for a future event, let us know.
This field is only just starting to emerge, and it stretches across academic, tech and policy fronts. This means that policymakers, journalists, educators, and consumers don’t have an easy way to keep up with key developments as they materialize. We’re in the early stages of addressing this by building an Evidence Hub — starting with a searchable collection of key AI benchmarks and leaderboards for important Humane Evals phenomena. We’ll be launching a prototype later this year. If you’d like to contribute to it, please get in touch at evals@humanetech.com.
We’re still in the early stages of our Humane Evals work. There are many unknowns, and we know that not every bet will pay off. But we also know that many other organizations and individuals are working on the same agenda. Center for Humane Technology is excited to be part of the collective effort to accelerate and highlight this vital work.
If you’re looking to collaborate on one of our projects, partner with us, fund our work, or share data you’d like the world to use, you know where to find us.
All of this work is supported by the guidance and advice of our Humane Evals Steering Committee, consisting of thoughtful and innovative experts at the frontiers of this new field. We’d like to thank:
Privahini Bradoo, Co-Founder and CEO of Plank
Stéphie Herlin, Co-Founder of Korabench.AI
Tasha McCauley
Jared Moore, researcher at Stanford HAI
Mitch Prinstein, Co-Director of the Winston Center for Technology and Brain Development
Conrad Stosz, Head of Governance at Transluce
Gayle Karen Young, CHT board member
We have become very good at measuring what AI can do and surprisingly bad at measuring what it does to us. Perhaps the missing benchmark isn't model capability, but human capability after prolonged dependence on the model.
I'm excited to learn of your interdisciplinary approach to AI. The nonprofit sector that focuses on community safety net issues is in "learn while you drive" mode to make the best uses of AI for workflow improvements WITHOUT compromising confidential data. Thank you for taking this on!