Blake Richards · @tyrell_turing
1006 followers · 347 posts · Server fediscience.org
6/10) We find that activations across different layers have an #eigenspectrum that follows a #powerlaw. Furthermore, well-defined intervals exist for the power law decay coefficient, α, where models exhibit excellent #OoD #generalization! 📈🎉🥳
#neuroscience #deeplearning #ml #ai #generalization #ood #powerlaw #eigenspectrum
Blake Richards · @tyrell_turing
1006 followers · 346 posts · Server fediscience.org5/10) In our paper, we study the #eigenspectrum of #DNN representations trained across different loss functions, architectures, and datasets and assess the corresponding out-of-distribution (#OoD) #generalization performance.
#neuroscience #deeplearning #ml #ai #generalization #ood #dnn #eigenspectrum
Blake Richards · @tyrell_turing
1006 followers · 344 posts · Server fediscience.org3/10) Driven by recent 🧠 findings in the #visual #cortex, we propose using the slope of the #eigenspectrum decay of the representation #covariance, termed α, as a measure of representation quality for #SSL model representations.
#deeplearning #ml #ai #ssl #covariance #eigenspectrum #cortex #visual