Currently watching a #DeepLearning experiment I'm running. I have two identical networks. One is running standard #backpropagation. The other is being trained in two segments, with the second half using standard backprop and the first have being trained with a #SyntheticGradient. The synthetic gradient version is kicking standard backprop's ass, and it feels like a magic trick.