Django admin is great for administrating a site, but chances are you won't want to run that on your "big serious production site" for a few reasons.
This is pretty easy to do, all your apps will be stored in source control (or something more exotic) so pull those apps down. Leave out the apps you don't need. Perhaps you've got a public facing specific app that pulls in public specific things, sets up context processor or specific auth. backends, all things you don't need in the admin site.
This could be set up on the same server, or on a different server as long as it can reach the database.
There's no limit to the number of password attempts on the admin site. So you could dictionary attack the admin site if you really wanted to get in. Moving it to a separate instance isn't necessarily more secure, unless you set up an appropriate policy.
For example: all requests to admin.yoursite.com are internal only, or restricted by IP. Those sorts of policies are very easy for system admins to setup and maintain.
Knowing that pretty much no matter what the end user does, they can't access django-admin is a big comfort.
Admin can be a bit slow if some queries aren't optimized, you'd be really annoyed if someone browsing the admin site took down your public site. Likewise the public facing site might have options that are unnecessary on the admin (such as the afore mentioned context processors).
At DjangoSki I was chatting to Jakob about this and found we've both also run admin on ready-only database slaves. So it's not really an admin, just a data browsing site.
For a big site you will likely have quite a few things in front of a site. Your media is likely on a CDN. You've likely got a load balancer and a caching server in there. All things that make your site go nice and fast. You don't need any of those for the admin. Again, route the requests differently.
There's apps that you might want to run on one instance and not the other. At one time I ran a SQL logging app on the public site over a few hours to track down some performance, that was a public specific app. It was easy to turn that on the public site and not have to worry about logging admin queries.
You don't even have to limit yourself to just one admin. If you really, really, wanted to, you could have multiple admins, each accessing different apps. Django admin will only allow you to admin apps you have INSTALLED_APPS, so if you really wanted to you could have one admin for group X that only accessed app X. However if you creating lots of admins, you'll probably find better ways to do that.
Setting up admin to run separately is pretty easy and I'd fully recommend it for any project of a serious size.