"Deontological and absolutist moral dilemma judgments convey self-righteousness" in U.S., German-speaking, and British participants (N = 1254).

In the Journal of Experimental Social Psychology: https://doi.org/10.1016/j.jesp.2023.104505

#ProcessDissociation #DecisionScience #psychMethods #moralPsychology #xPhi

#Civicbase brings upvoting and downvoting to preference measurement—but with a budget.

Participants can select or agree or disagree buttons (up to 7 times) to allocate a limited voting credits (that carry over to future studies?).

May reveal priorities that Likert scales and ranked-choices cannot.

https://doi.org/10.1002/aaai.12103

Presumably, this could be used for all sorts of preferences (beyond civics/politics).

#measurement #PsychMethods #openSource #decisionScience #poliSci #cogSci #gamification

How do we know what participants thought when we presented our stimuli?

#ProcessTracing can reveal what people saw (e.g., eye-tracking), consciously thought (e.g., concurrent think-aloud), etc.

Combining those two methods revealed:
(1) thinking aloud didn't impact gaze or word count
(2) retrospective think-aloud left out thoughts that were mentioned concurrently
(3) retrospective think-aloud introduced thoughts unmentioned concurrently

https://doi.org/10.1007/978-3-319-14956-1_5

#PsychMethods #CogSci #xPhi

Planning a longitudinal study? Here’s four questions you should ask:

🔹 How should time be scaled?

🔹 How many assessments are needed?

🔹 How frequently should assessments occur?

🔹 When should assessments happen?

Hopwood et al. (2022). “Connecting theory to methods in longitudinal research”:
https://doi.org/10.1177/17456916211008407

Author on Mastodon: @aidangcw

#Stats
#Statistics
#Methodology
#Psychology
#PsychMethods
#ResearchDesign
#LongitudinalResearch

New paper provides a history of “voodoo science,” which discusses the controversy surrounding Vul et al.’s (2009) controversial article “Puzzlingly High Correlations in FMRI Studies of Emotion, Personality, and Social Cognition.”

Five quotes follow: 🧵👉

🔓 https://doi.org/10.3390/socsci12010015

#MetaScience
#Neuroscience
#Neuroimaging
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#Fmri
#VoodooCorrelations
#UseNovelty
#MultipleTesting

Critical Metascience:

2022 has been a bumper year for what I’d call “critical metascience” - work that takes a step back and offers a critical perspective in the field.

My Top 10 papers of 2022 in this area are, in alphabetical order… 🥁 🧵👉

#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#SociologyofScience
#ScienceofScience
#PhilosophyOfScience
#PhilSci
#PhilScidon

1/12

Replicability and Theory:

“Our results suggest that many of the practices that have been proposed as a means to improve the replicability of psychological research—such as open data and methods…preregistration and Registered Reports…and basing conclusions on Bayesian inference…or p < .005 rather than p < .05…—do indeed improve confidence in replicability among our sample.”

Continued 🙂 🧵👉

#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon

@MarkRubin This is massively simplistic. Hypotheses include the criteria for delineating phenomena in need of explanation, satisfaction criteria for success, disciplinary standards and practices, and taxonomies of subjects under investigation. IMO.
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon

What’s a hypothesis?

“A hypothesis is not simply a guess about the result of an experiment. It is a proposed explanation that can predict the outcome of an experiment. A hypothesis has two components: (1) an explanation and (2) a prediction. A prediction simply isn’t useful on its own.” (Haroz, 2014)

Blog post: http://steveharoz.com/blog/2014/mysterious-origins-of-hypotheses-in-visualization-and-chi/

#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon

A “quietist” response to the replication crisis:

“The quietist approach proposes that we should just accept that it is in the nature of science that we get things wrong, and that this is particularly true with sciences in early stages of development.”

Bird (2021). Understanding the replication crisis as a base rate fallacy.

🔒 https://doi.org/10.1093/bjps/axy051

🔓 https://kclpure.kcl.ac.uk/portal/files/95454096/Replication_base_rate_fallacy_REV2.pdf

#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
#PhilScidon
@philosophy

Bad Stats / Poor Methods:

Qualitative study finds 39.8% of 548 psychology researchers believe that statistics and/or research methods are misused and/or misunderstood in the field.

Miranda et al. (May 2022). How do researchers in psychology perceive the field? A qualitative exploration of critiques and defenses. Collabra: Psychology.

https://doi.org/10.1525/collabra.35711

#Psychology
#Stats
#Statistics
#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis

No evidence of p-hacking in imaging research:

Analysis of 4,105 randomly sampled p-values finds no evidence of p-hacking in work published in over 100 imaging journals since 1972.

Rooprai et al. (2022): https://doi.org/10.1177/0846537122113941

#Stats
#Statistics
#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis

Do you have #MissingData ? (Yes, we all do). Do you worry your imputation model may not be correct? (Yes, we all do!) Check out this paper by colleagues and alum from #UCLA for a description of methods to evaluate compatibility of your imputation model! #Statistics #PsychMethods #QuantMethods https://doi.org/10.3758/s13428-021-01749-5

Looks like a great talk from Stephan Guttinger on Questionable Research Practices

“What should be abandoned is not the idea of questioning practice, but the idea that there is a class of questionable research practices.”

Slides: https://philstatwars.files.wordpress.com/2022/12/guttinger_draft_red.pdf

#OpenScience
#MetaScience
#MetaResearch
#PsychMethods
#ReplicationCrisis
#QRPs

“We are not only in a replication but an interpretation crisis, a crisis of theory building.”

Benjamin Krämer (@benjkraemer) (2022, November). Why are most published research findings under-theorized? In Questions of Communicative Change and Continuity.

🔓 https://www.nomos-elibrary.de/10.5771/9783748928232-23.pdf

#OpenScience
#MetaScience
#PsychMethods
#ReplicationCrisis
#ScienceofScience
#PhilosophyOfScience
#PhilSci
#PhilScidon
#Communication

“The appropriate conclusion based on significant social-psychology experimental findings could perhaps be characterized as ‘sometimes this happens.’….Although ‘sometimes this happens’ may be disappointing compared with establishing universal laws, perhaps the field should accept this with both humility and pride.”

Baumeister et al. (2022). https://doi.org/10.1177/17456916221121815

#MetaScience
#PsychMethods
#ReplicationCrisis
#PhilSci
#SocialPsychology
@socialpsych

@dp_moriarity Nice quote from the abstract:

"Results show that factor scores have (a) higher correlations with true scores (Δ = 0.02–0.04), (b) higher sensitivity (Δ = 4–8 percentage points), and (c) higher reliability (Δ = 0.04–0.07). Factor score performance metrics also have less sampling variability in most conditions."

Let's not just use sum scores!

#Psychometrics #PsychMethods #FactorAnalysis

Replicating experiments:

New preprint considers the “minimum viable experiment to replicate.”

“In this paper, we introduce the idea of a minimum viable experiment that needs to be identified in practice for replication results to be clearly interpretable.”

Devezer & Buzbas. (2022). Preprint: http://philsci-archive.pitt.edu/21475/

#OpenScience
#MetaScience
#PsychMethods
#ReplicationCrisis
#PhilosophyOfScience
#PhilSci
@philosophy

On the questionable use of “Questionable Research Practices” (QRPs):

“Practices labelled as ‘QRPs’ can be both beneficial and problematic for research practice and targeting them without a sound understanding of their dynamic and context-dependent nature risks creating unnecessary casualties in the fight for a more reliable scientific practice.”

Stephan Guttinger. Talk to be presented on 1st Dec: https://phil-stat-wars.com/3-tentative-schedule/

#Statistics
#OpenScience
#MetaScience
#PsychMethods
#ReplicationCrisis

New evidence that Bayes factors are misused in applied psychology:

“The way forward is not to ban Bayesian inference from our toolbox. Instead, more and better education on Bayesian inference is needed.”

Preprint: https://psyarxiv.com/du3fc/

#Psychology
#Stats
#Statistics
#PsychMethods
#Bayes