This will remove them as your friend. It should be noted that Xbox Live will not consider two people "friends" until both of them add each other or accept friend requests (a la 360).

A lack of must-have exclusives continues to stall Microsoft’s momentum, despite the Xbox Series X being a hardware beast.