Notorious Maine fisherman going back to federal prison