There were two terrible things here:
* Some nasty pointer-based casting
* Storing references to temporaries
Unfortunately, storing addrs of temps was happening through a helper
function, so VS won't warn here. I think I caught all the cases where
those functions were being used wrongly, but I can't be sure until
coverity runs on this commit.