In a bit of wind, kayaks will always turn to one side. The kayak often turns to the direction of the wind which is natural. Sailboats will do this also. This is known as weathercocking. You need to use your paddle to steer or rudder to counteract this turning. Here is an article about weathercocking:
Edging the kayak to the side will also make the kayak turn so as you paddle if the boat leans over to the right the boat will turn left and if the kayak leans a bit left then the boat will turn to the right. Kayakers without rudders will use the edging technique to help them steer their kayak. Here is a good video:
I hope this helps but at this time I am not sure there is anything wrong with your kayak.