If you’re panning to follow a train, ON and Mode 2 are the appropriate settings.
If you’re on a tripod, OFF. If you’re using fast shutter speeds (1/1000 s or faster, conservatively), it won’t help but nor will it hurt to leave it on. Mode 1 is general purpose (holding the lens steady or tracking an erratic subject), Mode 3 is like Mode 1 but it doesn’t start stabilizing until you press the shutter.
One thing to keep in mind with all lens image stabilization systems is that it takes about half a second for stabilization to fully kick in. IS is activated when you half press the shutter (or the AF-ON button if you’re using back button focus), but if you just mash the shutter button down the IS system can actually make the first few resulting images blurry until the system fully engages.