You'll want to start with your PC. If you're just starting out or don't know much about computers, buy a standard PC but make sure it can run demanding games. It's cheaper to buy a PC tower by itself.
Some decent specs are intel i5 3.2ghz processor or equivalent, GTX 550Ti graphics card or equivalent, at least 8gb RAM. These are all minimal, and would not be able to run all games perfectly.
Next, you'll need a monitor. 1920x1080 is the standard size for most monitors, but the higher the resolution, the more powerful the graphics has to be to run a game at decent frames per second.
You'll also want a gaming headset or just any headset with a microphone. Unless you're a little kid, most people appreciate when somebody has a microphone because it makes gameplay much more fun.
Now you need a mouse and keyboard if your computer didn't come with them. Make sure they are comfortable and the keyboard doesn't have any extra buttons near the WASD keys since they get in the way.
Finally, you need a variety of multiplayer, co-op and singleplayer games. Some of the basic games you should have are Battlefield 3, Crysis series, Fallout series, Assassin's Creed series... (cont)
...Garry's Mod, Far Cry series, RAGE, BatmanArkham City, Borderlands 1+2, Team Fortress 2, Skyrim. These are just a few common games, there is a huge amount of great games, many of them free.
Some optional things you could get are a specialized mouse with customizable buttons (commonly used for MMOs), a gaming keyboard and a 3D monitor since many new games support 3D.
Try to stay organized, don't tangle your wires and keep all your games in one place. PCs have much better graphics, smoother gameplay and a better community than consoles. Have fun!
You should also consider using Steam. It's a great software used to buy and download games online, add friends, join clans and do many other things. You can download it at Steampowered.com