As far as I understand, the opposition stands as follows (comes out of Freud's Three Essays on the Theory of Sexuality, which I have recently had the pleasure of "professoring," pace Leigh):
Perversion means that the sexual instinct has found "sexual" ways to express itself, which have not been affected by the resistances of guilt, shame, disgust or morality. Perverts adopt sexual objects or sexual aims outside the group of those included within the conventional determinations. Although they have expressed themselves "sexually," they are pathological insofar as they deviate from conventional morality. I.e. homosexuals are happy in their sex lives, but forced to live under unjust conditions.
Neurosis describes the state of the sexual instinct in which it has not found "sexual" ways to discharge itself, but, due most frequently to a transgression of a social more, instead created a symptom which discharges the accumulated, blocked sexual tension. The neurotic has internalized the resistances (listed above) which shape 'normal' sexual life, but essentially has increased their power to a pathological leve.
The pervert does what the neurotic wants to do. The pervert is punished by society, the neurotic is punished by himself. The rest of us, the 'normal' ones, we have a little bit of both the pervert and the neurotic in us.
I have doubts about what consitutes sexual ways of discharging sexual tension. Especially when we're talking about symptoms. It seems like those symptoms should be "sexual." I mean, essentially they are in their psychic meaning, yet they seem to have no physiological correlate?