Making a high-resolution screenshot for printed materials will always be fraught with peril. I learned that the hard way when I attempted to incorporate screenshots of Ableton Live into artwork for the cover of Eponymous 4’s In C. A simple Google search for "300dpi screenshot" yields a lot of advice, some good, some bad. (The first result in the search is actually incorrect.)
For an explanation of why making high-resolution screenshots are so difficult, this AskMeFi question goes into good detail. This technique yielded the best results for other programs but not for Ableton Live.
The developers of Live have gone to great lengths to make the user interface look sleek in Windows XP. All the text is anti-aliased — not a single pixelated character in the interface. (Except maybe the menus.) Those anti-aliased characters wreak havoc when upscaling Live screenshots from 72ppi to 300ppi (or dpi, since that’s what Photoshop calls it.)
After experimenting with a whole bunch of tools — including some screenshot utilities for gaming — my only course of action was to remove all text from my Live screenshot. Once the text was gone, changing the color mode from RGB to Indexed Color resulted in a smaller palette, and I could resize the screenshot. Of course, that meant if I wanted the text back in the shot, I would have to add it manually.
Then again, there really is no way to go from 72ppi to 300ppi without some clean-up at the end of the process.
Apple users probably don’t have a problem with this.