Aferrarme comes from the verb aferrar, meaning 'to cling to' or 'to hold on tightly'. It's a strong verb that conveys a sense of desperation or deep attachment.
In the context of the song, the singer uses it to express his struggle to let go of a beautiful but perhaps doomed relationship: "El aferrarme a esto tan bonito" (My clinging to this beautiful thing). It highlights his emotional resistance to ending the relationship, making it a poignant and relatable word.
Marco Antonio Solís writes this one from the losing end of a relationship he can't bring himself to end and can't make himself save. Right away he admits the problem: 'No sé decir adiós a alguien que amo' [I don't know how to say goodbye to someone I love], but he's already decided there's no other way out. Instead of ending things face to face, he hands her the goodbye and lets her decide what to do with it: 'Te dejo un "Hasta siempre" aquí en mi mano' [I leave you a "goodbye forever" here in my hand], then goes further and puts the whole decision on her conscience.
The verses are less about her and more about his own cowardice. He calls the relationship a mistake he kept holding onto anyway, 'El aferrarme a esto tan bonito' [clinging to something so beautiful], and flat out says he can't face her: 'Pero no soy capaz ni de enfrentarte' [But I'm not even capable of facing you]. By the time the chorus lands on the title line, 'Tu amor o tu desprecio he de llevarme' [Your love or your scorn, I'll take it with me], he's stopped trying to control how this ends. He just knows he has to leave with whatever she decides to feel about him.