July 19, 2012 — At the May 15th Atlantic States Marine Fisheries Commission (ASMFC) Menhaden Technical Committee meeting and in May 29th, June 7th, and 15th menhaden teleconferences, concerns were raised among fisheries scientists and committee members about the 2012 ASMFC Atlantic menhaden assessment update.
1. Background:
The results of the update have revealed several flaws in the assessment model. These problems were considered to be somewhat surprising, as the model has been used for many previous assessments and was recently peer reviewed during the 2010 benchmark assessment.
• A benchmark assessment is the only time when the model used to estimate the size of a stock is fully evaluated and changed and new data sources can be considered.
• Because the 2012 update assessment was a “turn-of the-crank” assessment, meaning that no changes to the model could be made or considered, the Stock Assessment Subcommittee (SAS) was unable to resolve these concerns.
Several scientific concerns have members of the stock assessment team questioning the update’s accuracy, and debating whether it should be used for fishery management purposes.
• The emergence of a retrospective pattern
• Inconsistencies in abundance indices
• Incorrect selectivity assumptions
The potentially flawed assessment came to the following troubling conclusions about the menhaden stock:
• The 2012 update assessment indicates that Atlantic menhaden may be experiencing overfishing and could potentially be overfished.
• It showed that fishing mortality was three times over the limit and that the abundance of the species was below its target value.
As a result of the concerns that have arisen about the accuracy of the assessment, in both the May and June meetings the Stock Assessment Subcommittee discussed recommending to the ASMFC Menhaden Management Board that a new benchmark assessment be conducted as soon as possible. Their recommendation will point out some of the issues that have been identified with the current model, and will be submitted to the Management Board on August 8, 2012.
2. Issues of Concern:
A “Retrospective Pattern” Emerges
A problem common to fishery assessments, including the menhaden assessment, is what is known as a retrospective pattern. A retrospective pattern occurs when the population and mortality estimates for a specific year (called a “terminal estimate”) are challenged by new data provided by subsequent assessments. When new stock assessments are conducted, the data they collect is compared with the estimates of previous assessments. If, when the data sets for these years are combined, the estimates produce an outcome that contradicts the previous estimates or is scientifically or mathematically impossible, then a retrospective pattern has occurred.
In the case of the menhaden assessment, the SAS tested the model by conducting successive model reruns that progressively ended the model run one year earlier, i.e. making 2010, 2009, 2008, etc. the terminal year. The results of these new terminal year estimates were compared to the results of the initial run of the entire time series (called the “baseline run”). The resulting estimates from the retrospective analysis showed that over the past five years, the Beaufort Assessment Model, a forward projection stock assessment model currently used (named after the location and lab where it was first used by NMFS), was consistently overestimating the rate of menhaden harvest and underestimating the menhaden population.
With the addition of the 2012 update assessment results, it was found that from 2006 to 2010, the menhaden assessment model showed a pattern of underestimating the population size and overestimating the fishing mortality rate. This indicates that the model being used for menhaden assessments may be too pessimistic, implying that more fish are dying than actually are. These results suggest that when the next stock assessment occurs, the actual mortality rate might be lower and menhaden abundance might be higher for 2011 than is currently established by this “turn-of-the-crank” update.
To explain the retrospective pattern, scientists look to the assumptions on which the assessment model is based. Each assessment relies upon a varied set of assumptions that range from assumed mortality and natural mortality to catchability. The accuracy of the assessments is dependent upon these assumptions being met. When one of the assumptions is not met and the assessment proceeds as if it had been, it can generate inaccurate data, resulting in a retrospective pattern and inaccurate estimates. If a retrospective pattern is seen in an assessment, the model used needs to be examined to locate the assumption or assumptions that have not been met. That was not done for the 2012 assessment, however, because it was a “turn of the crank” and the Stock Assessment Subcommittee does not have the ability to change the model or investigate its assumptions.
As a general rule, according to guidelines developed by NOAA’s National Marine Fisheries Service, when an assessment displays a retrospective pattern, it is considered inaccurate and should not be used for fisheries management purposes. Chris Legault, Chair of the NOAA Retrospective Working Group in their January 2008 Report wrote: “A strong retrospective pattern is grounds to reject an assessment model as an indication of stock status or the basis for management advice.”
The resultant inaccuracies, due to retrospective patterns, can spur further complications. In particular, when an assessment with a retrospective pattern is used to generate an estimate of a species' stock size, the future projections of stock size will also suffer from that defective data. In the case of this menhaden assessment, that means inaccurately low projections of the species' population size. If the total number of fish cannot be properly estimated, this makes it difficult, if not impossible, to make appropriate stock rebuilding plans and set fisheries regulations.
A proposal was given by the Stock Assessment Subcommittee at May’s meeting to start projections a few years before the terminal estimates were made, in hopes that this would decrease the influence of the retrospective pattern. The proposal had to be rejected due to the rules of the “turn-of the-crank” update assessment.
Inconsistencies in Abundance Indices
A fish abundance index is a number derived from collected data that represents the quantity of a specific species used in fishery assessments. The menhaden assessment uses two separate indices that are compiled to estimate the total population size: an index of older fish based on fisheries data, and an index estimate of juvenile fish obtained from shallow water sampling.
A concern was raised in the menhaden Technical Committee meeting that the separate trends observed in the menhaden indices were contradictory. The juvenile abundance index has remained stable over the last few years, while the adult abundance index has been increasing. If the proportion of older fish in the population is increasing, but juveniles are remaining constant, it can be logically inferred that fewer adult fish are being removed from the water.
However, the assessment asserts the opposite: that adult fishing mortality is actually increasing. To explain this inconsistency in the assessment, it is possible that either the calculations for the model used on the indices are inaccurate and producing faulty data, or that the claim that mortality is increasing is incorrect.
Incorrect Selectivity Assumptions
“Selectivity” refers to the likelihood of a species being caught by a fishery. Selectivity varies within a species based on factors such as age, size, or location. This variation is modeled on a selectivity curve, which quantifies and compares how susceptible to fishing different groups within a species are. A group with higher selectivity means that it is more likely to be caught than a group with lower selectivity.
The current menhaden assessment fails to account for the migration of menhaden in its calculations. After fish reach the age of four they begin to migrate north in spring and early summer to New England and Canadian waters. This migration corresponds with the fishing season, putting the older fish outside the range of the fishery, which is concentrated in the Mid-Atlantic. However, the model assumes that this large portion of the menhaden population on which the commercial fishery does not fish are still available to be caught at the same rates as younger fish. This leads to an overestimation of the selectivity of older menhaden.
The age selectivity curve used in the current assessment for menhaden plots each age against its corresponding selectivity value. The model used for this curve assumes that adult fish age three and older are equally likely to be caught by the fishery, and that these age groups have the same high selectivity. When this assumption is graphed, the vulnerability to fishing increases until the age of three, at which point the fish are assumed to be caught by the fishery. Here, the selectivity curve pattern plateaus at maximum selectivity, creating a flat top. For this reason, it is known as a “flat-topped” selectivity curve (View an example of a flat-topped selectivity curve from James T. Thorson and Michael H. Prager’s 2010 report in Transactions of the American Fisheries Society, “Better Catch Curves: Incorporating Age-Specific Natural Mortality and Logistic Selectivity” here).
However, this assumption does not reflect conditions in the actual fishery. Menhaden become increasingly vulnerable to being caught as they grow to three years old, and then become increasingly less susceptible to harvest. Fish under the age of one inhabit shallow waters and are not often caught by the fishery, while older fish are not available to be caught because they are in the northern waters during the fishing season. So, the true selectivity curve for the fishery would take a different shape. Because the fishery only catches a high proportion of the middle age group of one to three year old fish, the selectivity is higher for these age groups than all of the others. The increasing availability of menhaden up to the age of three, and the decreasing availability beyond that age creates a curve that is bell-shaped. This is known as a “dome-shaped” selectivity curve (View an example of a dome-shaped selectivity curve from the Thorson report here, age is represented on the x-axis and catch is represented on the y-axis).
Because the current flat-topped model assumes that the fishery is catching an equal proportion of fish over the age of three, while the fishery actually catches a declining proportion of these ages, the catch is probably overestimated. The use of a flat-topped model, as opposed to the more accurate domed model, could explain why the calculated mortality rate is so high. If the assessment assumes a flat-topped selectivity for the catch when the true pattern is actually domed, it would produce similar results to those seen in the retrospective pattern: an overestimation of fishing mortality and an underestimate of population size.
When the dome-shaped curve was tested on the model by the Stock Assessment Subcommittee, the outcome was similar to when the assessment had been previously adjusted to account for the retrospective pattern. It had a much higher biomass and a lower mortality rate than the original runs. This could be a possible explanation of why the retrospective pattern occurs. Notably, the peer reviewers of the 2010 assessment suggested use of a dome-shaped selectivity pattern in the southern (and by far the biggest) portion of the fishery to account for the population’s age structure.
3. Concerns of the Subcommittee
A number of members of the Stock Assessment Subcommittee are concerned about these issues, and are expected to outline them in their final report to the Management Board, along with a recommendation that a new benchmark stock assessment be completed as soon as possible.
Nine members of Congress also wrote a letter to the heads of the National Marine Fisheries Service and the ASMFC asking that the model be re-considered and restructured before management strategies were to be put in place. The letter stressed the importance of the menhaden fishery to the economy, and to other fisheries, noting that regulations should not be made unless they are based on the best available science.
However, the final decision will be made by the ASMFC’s Menhaden Management Board, and they could delay the Amendment 2 process until a new benchmark assessment is conducted, as the serious issues in the current model used could keep the assessment from being an effective fishery management tool.