At the Pleasure of the President

There's something about Attorneygate that I haven't heard anyone ask about. The defenders of the administration constantly point out that the US Attorney's "serve at the pleasure of the president" (anyone know the origin of that phrase, I can't find it) and he can fire them at any time for any or no reason. Ok that's true. So are they saying that Bush was displeased with them? Are they saying Bush was involved in their firing? If so, why? All I've heard is that Bush once mentioned to Gonzales that he's heard some US Attorney's haven't been prosecuting voter fraud cases. That's a big difference from being displeased enough to fire them.