Well if you zoom it on one keyframe and its the only keyframe it will become the default zoom.
What you need to do is create a keyframe at the start of the clip with the default settings. If you suddenly want it to zoom at the keyframe of choice, zoom it in but also one keyframe back and another default setting one.
Basically the system changes the zoom gradually over time, so if you have alot of space between the original and the zoomed keyframes it will slowly get closer to the zoomed settings as the keyframe is approach, to shorten the time and make it a sudden thing you just decrease the distance between the two
Hope this helps, i can clarify if needed