I have never liked Safari's two finger swipe back animation. It is unnecessary and slow, and forces the previous web page to be reloaded (this particular part will change in OS X El Capitan). Most people's internet connections will load a web page faster than the animation can complete, so it just adds unnecessary waiting to web browsing.
Usually these animations can be turned off in an application's preferences or by running a Terminal command, but neither of those things exists for this animation.
Luckily, there is another alternative.
Go to System Preferences, click Trackpad, and click the More Gestures tab.
The first option is Swipe between pages. Click its dropdown menu and select Swipe with two or three fingers.
With this setting enabled, you can continue swiping back with two finger's with an animation, or you can now use three fingers to swipe back without an animation.
However, there is one big difference besides swiping with an additional finger: the three finger swipe is in the opposite direction of the two finger swipe. To go back with a two finger swipe, you two finger swipe to the right; to go back with a three finger swipe, you three finger swipe to the left.
If you're expecting natural swipes and scrolling, the direction change is a minor inconvenience, but it drastically speeds up web browsing.
Other OS X applications also use the three finger swipe. Finder and App Store are two examples. Firefox and Chrome already have their own two finger swipes, but the three finger swipe can also be used. Let me know if you come across other applications.