Anyway, I keep meaning to write up a blog post on “falsehoods I have believed about measuring model performance” touching on #AppliedML issues related to #modelEvaluation, #metrics, #monitoring, #observability, and #experiments (#RCTs). The cool kids would call this #AIAlignment in their VC pitch decks, but even us #NormCore ML engineers have to wrestle with how to measure and optimize the real-world impact of our models.

You have a problem: you currently pick thresholds for model-based actions using some arbitrary heuristic.

Your solution: pick the threshold that maximizes expected utility (e.g. revenue, profit, ROI, …) instead. That’s the definition of the rational decision, right?

Hmm, for some reason you now seem to have several more problems.
#DecisionTheory #Optimization #rationality #AppliedML

Stolen shamelessly from MarcJBrooker's post of Lamport's "state the problem" memo over on Twitter, but worth discussing nonetheless.

So many applied ML papers follow the local publication styles - with good reason - but fail to explain the explicit limitations and settings of their models.

We want applied ML papers to be accessible to non-ml domain experts, but at what point do we omit too much?

https://lamport.azurewebsites.net/pubs/state-the-problem.pdf

#AppliedML #ScientificPublishing #TransparentML