With me, it's location. And by that I mean, well, yeah, where I'm putting it. Duh.
No. Seriously. I have a dwarf Ginko in one of my beds in the front. I've had it at least two decades now and it's probably 18' or so high, maybe 10' crown.
The variety I've had my eye on is called Crimson Queen. It handles sun well. It's a dwarf variety: 10x10 area in full growth. That may not be your thing. The leaf structure on those are a lot like my Podocarpus: it'll 'flow' well.