I prefer only to add nonfat milk to my coffee.

I prefer to add only nonfat milk to my coffee.

In the second sentence, the limiting modifier only is placed in front of the words it modifies: nonfat milk.

For more help, see sections 12a, 12b, and 12c.