I dont like fighting?

I have been called a punk alot cause i dont fight at all.i am a very good fighter(2 years of tae kwon do, kickboxing and regular boxing). People just mess with me for the fun of it and i get really mad. They think i wont do anything cause i dont fight. SHould i start fighting people or just prevent fighting until its for a good reason.P.S. I know alot of pressure points especially the ones that knock people out