I'm looking for dataset/metrics for estimating domain distance between pairs of research publications.

I'm thinking of sth like the patent classification system (used to estimate technological distance between patents: https://arxiv.org/pdf/1503.02373.pdf), but for research papers.

#metascience #scienceofscience

Some recent conversations reminded me of this long (~40k words, like a thesis) but excellent "Vision of #Metascience" by Michael Nielsen and Kanjun Qiu:

https://scienceplusplus.org/metascience/

It spells out the problems with the incentives, norms, funding practices, etc. in research, then presents many ideas (some refreshingly wild!) not just on how to improve these processes, but do so continuously.

I just attended the intro call to this year's #CrowdPreprintReview* and I didn't realise there was still time to sign up and so I'm shouting about it here!

Want to be involved in working out the future of #PeerReview!? Get involved!

I'm gonna be on the teams for #ComputationalNeuroscience and #MetaScience and I'm super excited (if you couldn't tell! 😅)

https://mas.to/@ASAPbio/110622073457008476

@ASAPbio @prereview @sciety

#PrePrint
#Neuroscience

* Slides and recording will be available

Was invited to give a talk to the @brainhack_school today. Because they are an experimental bunch I decided to do something a little different and present my thoughts on #metascience (with some classic #openscience content).

I had a great time and got some good feedback.

Slides from the talk, titled Do the Science You Want To See in the World, available at the #osf link below. They recorded it so you should be able to watch at some point.

https://osf.io/3f4e2/

Here's a short blog about our new project, which seeks to better understand the community's experience of Registered Reports peer review:

https://www.cos.io/blog/registered-reports-community-feedback

#Metascience #RegisteredReports #RegisteredReport #PeerReview #OpenScience

Quite enjoying this article on questionable #metascience practices, many of which are just questionable science practices. I think pretty healthy criticism, and sounds introspection never harmed anyone. The title a bit on the nose perhaps though...

By @MarkRubin

https://journal.trialanderror.org/pub/questionable-metascience-practices/release/2

#metascience #PeerReview #ResearchIntegrity #ScientificPublishing

📢Have you authored or reviewed a Registered Report?

✏️Share your experiences and help build a picture of how RRs peer review is implemented across journals:

https://registeredreports.cardiff.ac.uk/feedback

📊So far, authors & reviewers have given over 240 pieces of feedback across 54 journals - see our dashboards!

#RegisteredReport #RegisteredReports #PeerReview #Metascience #OpenScience

"'A Suspicious Science' [#OxfordUniversityPress] is divided into two parts. The first part describes the history and contemporary practices of empirical #psychology [and in] the second part of the book, I explore popular forms of #psychology such as #selfHelp literature and its reliance on magical thinking."

Find out more about Amazon's "#1 New Release in #CognitivePsychology" in this week's series about it on the #BrainsBlog: https://philosophyofbrains.com/2023/03/26/a-suspicious-science-the-uses-of-psychology.aspx

#PhilSci #CogSci #MetaScience @PhilOfBrains

How much should we trust experts expectations about the outcome of clinical trials?

"five studies ...suggest that trials frequently deliver surprising results to expert communities and that individual experts are often uninformative when it comes to forecasting trial outcome and recruitment."

However, *aggregating* experts' expectations may help.

https://doi.org/10.1353/pbm.2023.0006

#PhilosophyOfScience #Epistemology #Forecasting #MetaScience #Medicine #Science #Expertise

📊 Have you authored or reviewed a Registered Report?

We want to understand the community's experience of RRs peer review:

https://registeredreports.cardiff.ac.uk/feedback

✏️ Please spend 5mins giving your feedback

#Metascience #RegisteredReports #RegisteredReport #PeerReview #OpenScience

@FrederikAust @uzh #MetaScience #MetaResearch

New blog post on the problems with a recent machine learning for metascience paper!

Feedback & comments welcome!

@socialpsych

https://www.psychologytoday.com/us/blog/how-do-you-know/202302/can-machine-learning-solve-psychologys-replication-problem

#AI #Replication #Psychology #PsychologyToday #MachineLearning #Metascience

📢 Calling all Registered Reports authors & reviewers!

We built a website to understand the community's experience of RRs peer review.

https://registeredreports.cardiff.ac.uk/feedback

✏️ Please spend 5mins giving your feedback.

#Metascience #RegisteredReports #RegisteredReport #PeerReview #OpenScience

@karolina @pwgtennant If meta-science involves careful critical appraisal and an assessment of methodological and reporting quality, then bad science doesn't prevent meta-science from proceeding. Naïve analyses will admittedly be GIGO. #MetaScience #BadScience #ResearchQuality #CriticalAppraisal

@inton @ARC_Tracker

Boosted for my followers - see the Nature news piece above in this thread: "‘Disruptive’ science has declined — and no one knows why".

Don Braben has argued since the 1980s that funding mechanisms (peer review, dependence on track record doing exactly the same thing in the past, and vetocracy by committee) are the problem, and disruptive scientists like Planck and Einstein would not get funding today.

https://en.wikipedia.org/wiki/Donald_Braben

https://doi.org/10.1038/316401a0

https://doi.org/10.1016/S0378-4371(02)01065-8

#MetaScience #research #ResearchFunding #funding #academiaews piece

3/3 - As a suboptimal research practice, operationalisation biases can according to Reddy "lead to false-positive findings".

In our experience this bias can also lead to false negative findings, especially when multidimensional constructs are not specified as such.

Divergent effects cancel each other out.

#metascience
#SocialPsych
#socialpsychology

Pleased to see that the issue of operationalisation bias is finally being brought into the light:

"the practice of operationalizing a construct selectively, through the exclusive employment of one or very few variables"

https://psyarxiv.com/prqyt/

It's an issue that led us to challenge a highly cited meta-analysis in Psych Bull on procrastination that made erroneous claims about procrastination not being linked to perfectionism.

1/2

#metascience

@MarkRubin Thanks so much for sharing! Still, I'd love to see more engagement between the #metascience crowd and (particularly feminist, decolonial) #sts social & cultural studies of science & technology - instead of that "let's ignore each other lest we inadvertently start rehashing the science wars". There'd be so much to learn from each other and perhaps even to fight together for.

Just enjoyed a very thought-provoking talk by @AndyPerfors on metascience. He sketched a framework for seeing science as an information system, since it is a toolkit for searching through an "idea space" to find trustworthy information. As a librarian, a lot of this resonates with me! 🤓

#AIMOS2022 #Metaresearch #Metascience #InformationSystems

Today we're launching Registered Reports Community Feedback - a site to better understand authors' and reviewers' experience of the Registered Reports peer review process:

https://registeredreports.cardiff.ac.uk

The broad goal is to collect data regarding how well various aspects of the Registered Reports process are implemented across academic journals.

#Metascience #RegisteredReports #RegisteredReport #PeerReview #OpenScience

1/5