Feds told to return nearly $650k to fishermen