I mean, I think we have enough to figure out before racking up more research debt (relationships, time, money, emotional, intellectual, etc).

#EvaluationFails #LLMs

Hearing evaluation professionals - again - fail to address fundamental problems with LLM/AI systems in the interest of efficiency or curiosity hurt my soul.

Why are they:

supporting the use of stolen data?

supporting exploitative labor practices?

putting participants' data and identities at risk?

so committed to adding another opaque and inequitable layer to our work?

I asked this during a VSA web chat today. Crickets. Then, "but efficiency! I'm curious."

#EvaluationFails

Is it forever my curse to find (and many times try to mitigate) terrible eval and strategy in museums, nonprofits, science centers, etc?

Reading some $$ reports from Ball State, conducted by PhDs. They're so stunningly bad I'm embarrassed on their behalf. Bad from every possible perspective - design of the study, analysis, interpretation, data visualization, theory...

AND these studies were done with vulnerable populations (facedesk intentifies).

#evaluation #EvaluationFails #WTF

@curatedjenny
The study gives every indication that it wasn't participatory or equitable in any perceivable way, so it's centered even more on the organizations and staff involved.

It's reductionist, at best. Certainly exclusionary.

The graphics are nice?

This kind of thing is deeply frustrating to many evaluators that give a damn about people and recognize complexity. Like, I completely understand your (@curatedjenny) response!

#evaluation #RealEvalTalk #EvaluationFails

What is my community? How should I frame that idea for this survey? Do I only get to choose 1? Or are all my communities monolithic?

And those options...

#EvaluationFails

Where to begin...

#EvaluationFails

What are we to do when state arts agencies can't design a valid survey? I mean, the purpose of their study is to inform their 5 year strategic plan!

I suppose it's entirely possible that a consultant designed it for them.

Doesn't change the sadness about the situation - just who gets most of the blame, I guess.

#Evaluation #EvaluationFails #Accountability