Just getting a name and phone number isn't a referral. Instead, you have to turn those worthless names and numbers into real introductions.
Using the phone to connect with prospects in a manner that interests them, captures their attention, and that they welcome requires using a more sophisticated process than just picking up the phone and making a cold call. Here's the system I use that changes my calls from cold calls to welcome introductions.
If you are using a traditional referral generation method, you're not getting the quality of referrals you want because the method you are using simply creates more problems than it solves.