This will remove them as your friend. It should be noted that Xbox Live will not consider two people "friends" until both of them add each other or accept friend requests (a la 360).