Wittgenstein's philosophy of mathematics has long been notorious. Part of the problem is that it has not been recognized that Wittgenstein, in fact, had two chief post-Tractatus conceptions of mathematics. I have labelled these the calculus conception and the language-game conception. The calculus conception forms a distinct middle period. The goal of my article is to provide a new framework for examining Wittgenstein's philosophies of mathematics and the evolution of his career as a whole. I posit the Hardyian Picture, modelled on the Augustinian Picture, to provide a structure for Wittgenstein's work on the philosophy of mathematics. Wittgenstein's calculus period has not been properly recognized, so I give a detailed account of the tenets of that stage in Wittgenstein's career. Wittgenstein's notorious remarks on contradiction are the test case for my theory of his transition. I show that the bizarreness of those remarks is largely due to the calculus conception, but that Wittgenstein's later language-game account of mathematics keeps the rejection of the Hardyian Picture while correcting the calculus conception's mistakes.