Adjust the windvane tilt
If there is no internal friction it may help to increase the angle of the windvane axix from 20 dgs (leading edge of the windvane blade parallel to the mast) to high angles (leading edge of windvane at 10-15 dgs to vane mast). This may help especially on running courses (wind from behind).
When you increase the angle, the travel gradually decreases. You can try taking the angle to almost 90 dgs. If you were to increse it to 90 degrees there would no longer be any vertical movement.
(Mr. Vane and Stainless WALT only)

Connection chain/tiller
It may also be the case that the boat moving down waves causes a change in direction that can not be corrected quickly enough.
This may be solved by moving the chain hook closer to the boats rudder stock/hinges. See "pendulum rudder stays at the surface" for more details.

Internal friction
Check if the windvane moves freely and smoothly. If it jerks there is probably friction in the path between the windvane and the vertical rotation of the pendulumrudder. Check the next page.
If there is no friction in the system, also check the rudder arrangement for friction too.