Rasé literally means "shaved" or "shaven." It's an interesting word because of its double meaning and how it plays into the song's title and narrative.
In the context of the song, Gainsbourg uses "pas rasé" (not shaved) to describe the protagonist's disheveled appearance, implying a certain carelessness or perhaps a night of debauchery. However, the word also appears in the phrase "en rasant les murs" (skimming the walls), which means to walk close to the walls to avoid being seen, suggesting a sense of shame or wanting to be inconspicuous. This clever wordplay adds depth to the lyrics, making it more than just a simple description.
Gainsbourg turns a bad night into black comedy. Unshaven and restless, the narrator drops in on an old flame — 'Par hasard et pas rasé, j'rapplique chez elle' (By chance and unshaven, I show up at her place) — only to catch her with someone else, 'un para, le genre de mec qui les tombe toutes' (a paratrooper, the type of guy who scores with all of them). Instead of a scene, he takes a walk, and the walk turns morbid: 'Je saute de tombe en tombe' (I jump from grave to grave), pacing a cemetery until the ache passes.
By the second verse the shrug has set in — 'Toutes les mêmes après tout, cette fille je m'en fous' (They're all the same after all, I don't care about this girl) — and the title's own pun closes the loop: a song that opens 'pas rasé' (unshaven) ends on the single word 'à raser' (to shave), as if getting over her is as simple as picking up a razor.