NativeHoustonian, Andrew Saxe, who used to work on White Collar, trying to keep it under budget, gave us a great insight on production costs.
Driving scenes - expensive and difficult to shoot
Exterior scenes - they need to contract with the owner of the building to shoot there (they pay a fee). White Collar has many outdoor scenes that showcase the best of New York City. If weather is bad, and they need to change their schedule, it costs money.
Interior scenes - again, they have to contract with the owner and pay a fee.
Avoid night scenes - causes longer days which means lotsa overtime for the production crew.
They provide meals for the entire staff, free of charge, all day.
They have trailers for many things, the wardrobe department has it's own, plus trailer dressing rooms. All of these trailers have to be driven to their destination by union drivers.
New York City is expensive. The unions make much higher wages, then say some other city. Food and lodging is much more expensive (guest stars may want higher pay, each week the writer flies to NY, the writers are based in LA). Jeff Eastin negotiated a tax credit with the city to get a break on the taxes. Chicago recently opened up a 1.3 million square foot sound stage to attract series, movies, etc. They also give a generous tax break. There has been a huge boom in what gets filmed in Chicago because of this (two of the Transformer movies were partially filmed here).
Many series are not filmed where they are supposed to take place. Covert Affairs is filmed in Toronto, not DC. Graceland is filmed somewhere in Florida, Suits is filmed in Toronto and not New York City.