Hello #statstodon! A reviewer asks me to perform a post-hoc #PowerAnalysis. I know this is generally not advised because if you replace the a priori effect size by the effect size measured in the experiment, this will introduce an erroneous relationship between the significance level of the test and the measured power.
… but does that mean that there is no proper way of measuring power retrospectively? For example, if you refrain from using the measured effect size and instead simulate a range of “a priori” effect size unrelated to the results of the test, then the dependency of the power to the significance level should not happen?
#stats #statschat @lakens

#statschat #statsdon #stats #rant
While not totally accurate, do you think if we start saying "if you're wrong, the p-value is random" would improve understanding of why they provide little info?

#statschat #statsdon #statistics #statistics

Do you know what Type M error and Type S error are?

(see Gelman & Carlin, 2014)

#statschat #statsdon #stats

What is the best way to "correct" for optional stopping (data peeking)?

I understand if I plan for a sequential design how to control the alpha, but what if someone comes to me saying "Oh, I looked at the data last week and the p-value was still big, I'm putting in for more funding to get more people"?

How do you correct the conditional probability?

interesting (free but sign-up required) seminar at MRC Cambridge in a couple of weeks

snippet from the abstract:

"How does a method go from ‘neat idea’ to mature and trustworthy, so we can say when it should be used or avoided? "

more info at https://www.mrc-bsu.cam.ac.uk/event/bsu-seminar-phases-of-research-for-statistical-methods/

#stats #statschat

The sampling distribution of the sample proportion of even numbers for 10,000 random samples of 35 squared digits (AKA Frankensquares). My students made their own unique sets of Frankensquares using some "random" numbers from their lives and some rolls on a d10 die. #statschat #APStats #mathchat #MTBOS #rstats

#statstodon #statschat
In the absence of a Registered Report how would one publish, say, a study on an improved measurement if you find in a Bland-Altman plot (Turkey mean-difference) the error/bias is too big? Namely, your "new" measure isn't good enough.

What would you do?

#statstodon #statschat
In the absence of a Registered Report how would one publish, say, a study on an improved measurement if you find in a Bland-Altman plot (Turkey mean-difference) the error/bias is too big? Namely, your "new" measure isn't good enough.

What would you do?

p-values higher than your alpha (say, 5%) are not to be taken as "evidence of no effect". Everyone with me so far?

OK, now explain why this is exactly how almost all mediation analyses, multivariate regression models, SEM, fMRI papers, etc. use them! 🤷‍♂️

#statsdon #statschat

#statschat Explaining statistical power seem to always be difficult. The two best options I've found is either by focusing on the (1) per study variability in p-values (as illustrated by the @thenewstats p-value dance) or (2) long term probability of detecting an effect (as illustrated by @lakens p-value distribution app)

https://www.esci.thenewstatistics.com/esci-dances.html

https://shiny.ieis.tue.nl/MOOC/assignment_1/

I wonder which people feel works best:

Statistics educators, come out come out wherever you are!

#statschat

#statschat #rstats #statsdon
Why don't ppl ever apply correlation corrections when they report results for single studies? e.g. attenuation and range restriction.

Is there some fundemental issue with these approaches, or is it just that most don't know they exist?

(I have a similar pet peeve with ppl never using tests for difference in correlations, like Steiger's Z, and just claiming difference)

#statschat Let's say you plan to do a replication study. How do you decide on the effect size of your new study given what you know about publication bias, type M error, etc.?

(poll only allows 4 options, so feel free to comment others)