There are tons of web servers online, however it is a little tricky when it comes to picking the right one. It depends on your bandwidth requirement, the amount of space needed, uptime and the most important of all; your budget.
Now that you've decided on your domain name, and which hosting service you're going to use, it is time to call HostGator and start the process. In case you're wondering, your website does not reside on your computer, instead it is located at HostGator. The support people at HostGator will walk you through the steps necessary to set up your website. They will take all necessary information, name, address, etc. and then ask for your domain name. They will then ask you what platform you want to use. Again, we strongly suggest you use WordPress.
HostGator will set everything up for you in thirty minutes or less. If you're concerned about the cost, don't be. A domain name costs roughly $10 a year, and your hosting service is approximately $5 a year. That's it. I would however recommend that you upgrade to a private registration for a small additional fee. And that's it.
Going about this is pretty simple. Purchase your podcast domain from a place like DreamHost or NameCheap. Then host it on one of these sites as well. If your podcast attracts heavy traffic, you may also want to consider storing website it with Amazon S3 or another storage service.
Many small business and personal sites are not going to get a lot of traffic. For such sites, it does not make any sense to spend hundreds of dollars on a dedicated web host like Rackspace or Media Temple. That would just be wasteful, as it isn't necessary.
Either way you go, remember that unlimited space shouldn't be the only draw when you pick a host or platform. Larger sites may cost a couple of dollars more per month, but their capacity and stability may be larger as well. If your blog has a sudden increase in traffic, the last thing you want is for your website to crash because your web host's servers are too slow to keep up. Either way you go, make sure you create an enjoyable, quality blog in the end.