A married man declared his love for me

I have been a friend to a man for about a year. He is married and almost 20 years my senior. Since he got sick, his wife doesn't pay much attention to him. He used to drink alot but since he has developed health issues related to drinking, he does not drink like he used to, but she still does. It is nice to have intelligent conversation with someone. The other night he called me and told me he is falling for me hard. This makes me feel very uncomfortable. I am not looking for a romance with this man. After all he is married and I just want a platonic friendship. He came over today and dropped off a necessity, and asked if he could call me sometime this week. I said it's not a good idea; you are married. Did I upset you with what I told you the other night? Yes, you did I responded. You are married, and I'm not looking for anyone most especially someone elses husband. He seemed very hurt by this. Now I'm thinking o, well you should not have allowed yourself these thoughts. Now I am uncomfortable to be around him. We usually meet on Sundays at a local restaurant where I pass out my Avon books. We have been going out to eat on Fridays, and he does want to see me on Friday. His wife doesn't go with him anywhere, so I have felt badly for him. I think he needs a good male friend instead of me. I just do not know how to break this off. I feel very wrong, even though I am not the one doing anything wrong in the least. What should I do about this? Is it ok to still see him occasionally or just stop this altogether? What would you do in this situation?