Here are some helpful solutions for the common causes:
Since muffins and other quick breads should not be overmixed, dry ingredients should be mixed together completely to insure proper blending of ingredients before they are combined with the wet ingredients. Use a fork or wire whisk for this step. If the dry ingredients are not well mixed, there can be pockets or lumps of ingredients (especially leavening) that can lead to bitter or off flavors.
Using old ingredients (rancid nuts, "old" shortening) will cause muffins to taste old or have an "off" taste. Purchase new ingredients if those on hand are questionable. Date ingredients on outside of container to help you remember how long they have been stored in the cupboard. Make sure all ingredients are stored in a cool, dry place.