21st December 2013
I was just asked this question and referred to Phoronix: Ubuntu 12.04 LTS: 32-bit vs. 64-bit Performance. Phoronix is this excellent web-site run by Michael Larabel.
Below is a chart from 2012. Not all charts look this astonishing!
This was also true in 2006, see below compilation times.
So the answer generally is 64-bit. There are of course some shortcomings, like possibly slightly larger binaries, or some binaries being slower, but in general 64-bit is the way to go, regardless of the memory size in your computer. If you want to address memory above 2 GB, then of course you have no real choice.
Added 11-Mar-2016: 64-bit can be more power efficient, see phoronix: Using 64-bit Linux Software To Reduce Power Consumption?
Added 18-Feb-2022: Similar findings for ARM, see phoronix: Further Investigating The Raspberry Pi 32-bit vs. 64-bit Performance