For sure don't be driving around - if you feel like you may want to be more face to face, look up Live Switch. You can do recorded facetimes and AI records it. You can even send them the link to upload it on their own. It's such a time saver, it's a no brainer.
On the giving advice, I've always been of the mindset to provide as much value as possible without expecting reciprocation. Will you have phone calls like that one here and there? Ya definitely but look at it as a good chance to practice closing people better. In the long term though, you will get more jobs than not and you will have a great reputation. If it really bugs you, ask them to give you a google review for the advice. 99% of people would say yes.
If you are really concerned with your time and really want the site visit, just be more ambiguous on the phone and tell them what you charge for a site visit ahead of time. I would probably argue you would rather tell that person on the phone that info then just getting $150 or whatever you charge just to tell them to their face then have them not use you. You want the job, not the site visit.
