“I’m not racist. I have Black friends.”
“How can you possibly suggest that I’m homophobic? I have a lesbian friend.”
When some of us feel that we are accused of being prejudiced, we can give a response along these lines. We defend ourselves against the accusation of prejudice (whether real or perceived) by pointing out that we have a friend or friends who are of the race, sexual orientation, gender identity, religion, etc., that we are accused of being prejudicial toward.
This language does one thing: it uses the people you call friends as a defense against an accusation of prejudice, often without the permission of said friend or friends. This is problematic on a number of levels.
For starters, the friend(s) you’re using as a defense often have no say in whether they are actually okay with being talked about and used in such a way. Given that fact, it is unfair to put friends in the middle of a controversy surrounding your potential prejudice. Your friends didn’t do anything to merit being in the middle of a controversy of yours, so the right and compassionate thing to do is to, well, not put your friends in the middle of one of your controversies.
Even if said friend(s) were okay with being talked about in that way, the “I’m not racist” or “I’m not anti-Semitic”, comments don’t do anything to address the form of prejudice being talked about. Saying that you’re not a racist usually does nothing about the racism that does exist in our society. Saying that you’re not sexist does nothing about the sexism that does exist in our society. All it does is attempt to convince yourself or others that you are not prejudiced in a particular way.
If anything, the “I’m not ____” comments are sometimes used to defend a word, phrase, or action that is prejudiced. I’ve read people say that that “most Blacks are lazy” (not making this up), an overtly racist comment, and then defend their racism by saying that they have friends of color. I haven’t seen this happen in my conversations too often, thankfully, but when it has happened, it has been disgusting.
Finally, your friends are a poor defense against prejudice because you can have friends of a particular group and be prejudiced toward said group at the same time. Albeit, if you’re prejudiced towards a group that a friend is a part of (for example, if you struggle with ableism and your friend is physically disabled), then that likely hinders your ability to be a good friend.
All in all, I would strongly recommend against using your friends as a defense against accusations of prejudice. It does no favors to you, your friend, or the cause of reducing prejudice in our world. You’re better off responding to those accusations, whether real or perceived, with self-reflection, signing petitions, and/or donating to causes that address the prejudice you’re accused of.
 Sometimes, with self-reflection, you might realize that something you didn’t realize sounded offensive to you was offensive to those around you.