What's the difference between microtask platforms like #mTurk or Prolific and market-research panels?

CloudResearch recently published an #openAccess paper that explains. (The paper also includes a step-by-step tutorial for using their "Prime Panels" in conjunction Qualtrics). 

https://doi.org/10.1177/25152459221140388

#humanSubjectsResearch #researchMethods #surveyMethodology #Psychology #CogSci #xPhi

For the #polisci & public opinion crew:

In psychology there are papers that tally what types of samples are most often used in the field (E.g., XX% college students, YY% Mturk etc). Are there any examples of this from your own fields?

Essentially, I'm trying to put some approximate numbers to the different norms in types of sample that I see

@politicalscience @socialpsych #research #sample #Mturk

A side question, given the high accuracy of classification of human generated vs synthetic text in this paper: if mechanical turk workers are mostly from certain countries, could the authors be training their model to recognise the style of English writing typical of those countries as "human"? Not that it would invalidate their results, but it could impact followup studies. #LLM #MechanicalTurk #ChatGPT #MTurk

@skry @mosheroperandi

The description of mturk:

'Amazon Mechanical Turk is a crowdsourcing website with which businesses can hire remotely located "crowdworkers" to perform discrete on-demand tasks that computers are currently unable to do as economically.'

LOL😂

I used to be a #mTurk crowdworker. Back when I first had to quit work and became totally homebound several years ago due to disability, every 10-cent job mattered

At the time, goal was to earn cable & internet bill via mTurk jobs

RT @turkopticon

It's time to recognize the challenges we face, take into account the autonomy & well-being of those working on #MTurk & create solutions that ensure the fairest treatment of Turkers worldwide.

Next Wednesday @ #Mozfest
:undefined:https://mozillafestival.org/en/tickets
:undefined:https://schedule.mozillafestival.org/session/U7L8W8

Next was an engaging talk by @KassemSarrah on #work and alienation in the platform economy at the @oiioxford. This is a multi-faceted analysis of the fulfillment center and #mturk #workers at #Amazon, and while there's an explicit agenda to the analysis I found the distinction between place-based and virtual organizing compelling https://www.youtube.com/watch?v=M75-cRiY6rU (3/4)

RT from turkopticon (@turkopticon)

Are you curious about the challenges of virtual global organizing in the #GigEconomy?

Our session at #Mozfest on "Uniting the Amazon #MTurk Workers Behind AI" is in less than 3 weeks! Don't miss it!

🎟️: https://www.mozillafestival.org/en/tickets/
🗣️: https://schedule.mozillafestival.org/session/U7L8W8-1

Original tweet : https://twitter.com/turkopticon/status/1631308394052329473

RT @turkopticon

Are you curious about the challenges of virtual global organizing in the #GigEconomy? Our session at #Mozfest on "Uniting the Amazon #MTurk Workers Behind AI" is in less than 3 weeks!

Don't miss it!
:undefined:: https://mozillafestival.org/en/tickets
:undefined:: https://schedule.mozillafestival.org/session/U7L8W8

What conclusion do you draw, if any, when all (and only) the people who wrote in "non-binary" to the question, "What is your gender?" failed a survey's attention check?

Why would someone who puts in the effort to write a (vs. select a pre-written) response have been inattentive earlier in the survey?

I doubt that people reporting non-binary gender are generally less attentive on surveys.

So was it most likely a fluke?

#mTurk #CloudResearch #researchMethods #demographics #dataQuality

"Humans are adept at OBJECTNAV. Prior work [1] collected a large-scale dataset of 80k human demonstrations [2894 Hrs for $50k] for OBJECTNAV, where human subjects on #MechanicalTurk teleoperated virtual robots & searched for objects in novel houses."

#Mturk #AI #ObjectNav #PreTraining #Gigwork

#Open Access

https://www.semanticscholar.org/reader/71dd4cb05260b008de9d3fb0fb6aee9892b9959c

"Even ignoring questions arising from the characteristics and behaviors of study recruits, the conduct of the research community itself erode substantially the credibility of these studies' conclusions."

#Mturk #Fairwork #Gigwork

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4188289

Frustration and ennui among Amazon MTurk workers

"The impetus for this study comes from our desire to use the crowdsourcing platforms such as #MTurk in ways that recognize & respect the contributions (and indeed, basic humanity) of the people who complete our studies."

#CrowdSourcing #Gigwork #Fairwork #OpenAccess

#Behavior Research Methods

https://link.springer.com/article/10.3758/s13428-022-01955-9

mTurk HITs completed

Can you get them automatically added to a dataset when running a survey?

Sort of like you can automatically add Worker ID, Assignment ID, and HIT ID to the dataset with certain URL parameters.

#psychology #survey #research #mTurk #dataQuality

Was thinking while reading this of the nasty things my mom spouts about every Instacart driver who delivers her groceries. When modern ultra-luxuries go slightly wrong, I try to remember these app workers are making as little money as I do by gig working on Clickworker and Mechanical Turk

https://jacobin.com/2023/01/tech-friction-service-work-dystopia/

#gigwork #gigeconomy #technocapitalism #mturk #instacart #workfromhome

Was thinking while reading this of the nasty things my mom spouts about every Instacart driver who delivers her groceries. When modern ultra-luxuries go slightly wrong, I try to remember these app workers are making as little money as I do by gig working on Clickworker and Mechanical Turk #gigwork #gigeconomy #technocapitalism #mturk #instacart #workfromhome
https://jacobin.com/2023/01/tech-friction-service-work-dystopia/

@atriverside This new preprint doesn’t “refute” the earlier paper, neither did that one “trash” #MTurk. This new commentary points out that the claim by the earlier paper that only a handful of participants were human is misleading, which is true: many were discarded not for obvious cues that they were automated scripts harvesting easy money, but rather, because even when human, they failed the selection criteria. The point that many psychology papers based on #MTurk are likely poor remains.

Delighted to read this pre-print by Cuskley & Sulik refuting a recently published paper trashing #MTurk.
"Effective use requires specific expertise & design considerations. Like all tools used in research—from advanced hardware to specialist
software—the tool itself places constraints on what you should use it for. Ultimately, high-quality data requires careful experimental design. This is the responsibility of the researcher, not the crowdsourcing platform.
#Gigwork

https://psyarxiv.com/w7qy9

I deeply appreciate that Cloud Research keeps putting out white papers and pubs related to their #mTurk #research platform.

https://link.springer.com/article/10.3758/s13428-022-01999-x

Vanessa Williamson has another take on the ethical quagmire of #MTurk here: https://www.cambridge.org/core/services/aop-cambridge-core/content/view/B1BDFB1111B416DD0B71540CD6E7D94F/S104909651500116Xa.pdf/div-class-title-on-the-ethics-of-crowdsourced-research-div.pdf

Read the latest in a long line of repeated indications that #MTurk is an ethical and methodological disaster: https://csl.mpg.de/427800/webb_tangney__too_good_to_be_true_2022.pdf

TLDR: author pays research subjects minimum wage; subjects treat her survey with the same respect they’ve been shown.

To the author’s credit, she says as much: “Owing to the compensation structure, ‘workers’ have little incentive to invest the extra time and thought required by open-ended qualitative items.”