2.1 Total Period Estimate

In cluster sampling, we consider a 4-hour period divided into \(N\) clusters (i.e. videos) with sizes (i.e. durations) \(t_1, t_2, ..., t_N\) and counts (# fish) \(y_1, y_2, ..., y_N\). The total size of all clusters (i.e. total time of recorded video), \(T\), is thus known because we know the durations of all \(N\) videos:

\[T = \sum_{i = 1}^N t_i\]

If all \(N\) videos have also been counted, then we would know the total number of fish over the period assuming no fish were passing when videos were not being recorded (see the Limitations section below).

\[Y = \sum_{i = 1}^N y_i\]

The average number of fish passing per second, \(r\), would then simply be the total number of fish divided by the total video duration during the 4-hour period:

\[ r = \frac{\bar Y}{\bar T} = \frac{\sum_{i = 1}^N y_i / N}{\sum_{i = 1}^N t_i / N} = \frac{\sum_{i = 1}^N y_i }{\sum_{i = 1}^N t_i } \]

This value, \(r\), is known as a ratio estimator. Note that this calculated as the ratio of the means (and equivalently of the sums), and not as the mean of the ratios, which is used in the St2WRS method but would lead to a biased estimate here due to the variable durations of each video.

In reality, only a fraction of all \(N\) videos are counted. Therefore, we need to estimate the value of \(r\) and its variance based on a sampling of available videos.

Say we sample a set of \(n\) videos, which have durations of \(t_1, t_2, ..., t_n\) and counts of \(y_1, y_2, ..., y_n\). The estimated value of \(r\) (\(\hat{r}\)) can be calculated as the ratio of the sample means (\(\bar{y}\) and \(\bar{t}\)):

\[ \hat{r} = \frac{\bar{y}}{\bar{t}} = \frac{\sum_{i = 1}^n y_i / n}{\sum_{i = 1}^n t_i / n} = \frac{\sum_{i = 1}^n y_i}{\sum_{i = 1}^n t_i} \]

This makes sense because its simply the total number of fish counted from the sampled videos divided by the total duration of those videos. So it’s an average rate of fish passage per unit time overall all sampled time periods.

To quantify the uncertainty, the estimated variance of the ratio can be calculated by:

\[ var(\hat{r}) = \frac{1}{\bar{t}^2} \frac{N - n}{Nn} s_e^2\]

where \(s_e^2\) is the estimated error calculated using the sum of squares of the difference between each observed count (\(y_i\)) and the expected count based on the fish passage rate \(r\) multiplied by the video duration \(t_i\) (if video \(i\) has a duration of \(t_i\) seconds, the expected number of fish using the ratio estimator \(r\) is simply \(r \cdot t_i\)).

\[ s_e^2 = \frac{\sum_{i=1}^n (y_i - \hat r \cdot t_i)^2}{n-1} \]

Now that we have estimated the value and variance of the ratio estimator \(r\), we can use that to estimate the total number of fish over all \(N\) videos in the period and the associated variance.

The total estimated number of fish is simply the estimated rate \(r\) times the total duration of all \(N\) videos (\(T\)):

\[ \hat{Y} = \hat{r} T \]

The variance is then (note we can pull out the \(T\) since it is a known constant):

\[ var( \hat{Y} ) = var(\hat{r} T) = T^2 var( \hat{r} ) = \frac{T^2}{\bar{t}^2} \frac{N - n}{Nn}s_e^2 \]

2.2 Total Daily Estimate

After estimating the total number of fish and its variance for each period using the equations above, the total daily total estimate and variance can be computed by simply summing the individual periods of each day (note: this is the same approach as that for the St2WRS method in TR-25, which assumes the estimates and variances of the periods are independent).

For day \(k\), the estimated total number of fish overall all \(P\) periods (\(p = 1, 2, ..., P\)) is the sum of the total estimated fish during period \(p\) (\(\hat Y_{kp}\)):

\[ \hat Y_k = \sum_{p = 1}^P \hat Y_{kp} \]

Similarly, the variance is simply the sum of the period variances (assuming the periods are independent, and thus no covariance):

\[ var(\hat Y_k) = \sum_{p = 1}^P var(\hat Y_{kp}) \]

Lastly, a confidence interval for the total estimate daily count can be calculated using a t-distribution and the Satterthwaite approximation for the degrees of freedom (same as TR-25):

\[ \hat{df} = \frac{ \left( \sum_{p=1}^P a_p \cdot s_{e,p}^2 \right)^2 }{\left( \sum_{p=1}^P \frac{(a_p \cdot s_{e,p}^2)^2}{n_p - 1} \right)} \]

where

\[ a_p = \frac{N_p (N_p - n_p)}{n_p} \]

Thus the confidence interval for day \(k\) is given by:

\[ \hat Y_k \pm t_{\alpha / 2} \sqrt{var(\hat Y_k)} \]

2.3 Total Run Estimate

Similarly, the total run estimate (\(\hat Y\)) and its variance is the sum of the daily estimates over all \(L\) days:

\[ \hat Y = \sum_{k = 1}^L \hat Y_{k} \]

and

\[ var(\hat Y) = \sum_{k=1}^L var(\hat Y_k) \]

The estimated degrees of freedom are calculated by summing over all periods and days:

\[ \hat{df} = \frac{ \left( \sum_{k=1}^L \sum_{p=1}^P a_{kp} \cdot s_{e,kp}^2 \right)^2 }{\left( \sum_{k=1}^L \sum_{p=1}^P \frac{(a_{kp} \cdot s_{e,kp}^2)^2}{n_{kp} - 1} \right)} \]

The total run confidence intervals are then

\[ \hat Y \pm t_{\alpha / 2} \sqrt{var(\hat Y)} \]

2.4 Limitations

This methodology provides a sound basis for estimating the total daily and overall fish run. However, it does have the following limitations:

• It is assumed that no fish are passing when videos are not being recorded. This assumption requires proper configuration of the motion detection trigger settings such that videos are always triggered when at least one fish is passing. In reality, the trigger settings present a trade-off in terms of having too low of a sensitivity resulting in some fish not being captured on video, and too high of a sensitivity resulting in more videos being recorded that contain no fish. For this methodology to work best, it would be better to err on the side of too high of a sensitivity to ensure that all fish are captured on video.
• The estimated counts represent fish passage only during daylight hours (7 AM - 7 PM each day). Night-time hours could be easily included by adding additional periods to each day, which may require installation of lights at the dam so night-time videos can be reliably counted. Alternatively, the total daylight estimate could be doubled to represent a full 24-hour period. However, this would assume that the fish passage rates are similar during the night as during the day, which may or may not be true.
• The error associated with inaccurate video counts is not incorporated in the overall estimate of uncertainty. Some videos were counted independently by two or more users. The variability among these counts of the same video is a form of measurement error. When a video is counted multiple times, this methodology uses the mean count for that video, but it does not incorporate the variance as part of the uncertainty estimate. The impact of this variability is explored more below in the Sensitivity Analyses section. The current St2WRS methodology used by DMF also does not incorporate measurement error associated with volunteer counts.
• The estimated variance (uncertainty) of daily and total run counts assumes that the mean fish passage rate follows a t-distribution. However, because a t-distribution is unbounded, this can lead to the lower bound of the confidence interval having a negative value, which is not realistic. Alternative distributions such as the poisson distribution, which is commonly used to represent the number of events over a fixed interval of time and does not contain negative values, may yield more sensible results. However, this would require further research to determine if the fish passage rate does indeed follow a poisson distribution, and then how best to estimate the parameters of that distribution. This same issue also occurs in the St2WRS strategy.

2.5 References

• http://www.mass.gov/eea/docs/dfg/dmf/publications/tr-25.pdf
• https://en.wikipedia.org/wiki/Ratio_estimator
• http://www.math.montana.edu/jobo/st446/documents/ho5a.pdf
• http://www.math.montana.edu/jobo/st446/documents/ho7c.pdf
• http://wiki.awf.forst.uni-goettingen.de/wiki/index.php/Ratio_estimator
• http://home.iitk.ac.in/~shalab/sampling/chapter9-sampling-cluster-sampling.pdf

3.1 First Period (7 AM - 11 AM)

First, we’ll estimate the total run during a single period (7 AM - 11 AM) on this day (May 17, 2017). The following figure shows when each video was recorded and whether or not it was counted. Within this period, most videos were recorded before 9 AM.

This figure shows the number of fish counted in each watched video.

To account for the variable video durations, this figure shows the number of fish per second in each video.

In total, there were 129 videos recorded (total time = 3,695 sec), of which 36 were counted (total time = 1,067 sec) and 93 were not counted (total time = 2,628 sec). The total number of fish counted over the period was 184. Thus the average fish passage rate (\(\hat r\)) was 0.172 fish per second.

\[ \begin{aligned} N &= 129 \\ n &= 36 \\ T &= \sum_{i=1}^N t_i = 3694.9 \\ \sum_{i=1}^n y_i &= 184 \\ \sum_{i=1}^n t_i &= 1067.0 \\ \hat r &= \frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n t_i} = \frac{184.0}{1067.0} = 0.172 \end{aligned} \]

The total estimated fish run during this period (\(\hat Y\)) is thus 637:

\[ \begin{aligned} \hat Y &= \hat r \cdot T \\ &= 0.172 \cdot 3695.0 \\ &= 637 \end{aligned} \]

The standard error (\(s_e^2\)) is 25.54:

\[ \begin{aligned} s_e^2 &= \frac{\sum_{i=1}^n (y_i - \hat r t_i)^2}{n-1} \\ &= \frac{\sum_{i=1}^{35} (y_i - \hat 0.172 t_i)^2}{36 - 1} \\ &= 25.54 \end{aligned} \]

The variance (\(var(\hat Y)\)) is 7949.7

\[ \begin{aligned} var( \hat{Y} ) &= \frac{T^2}{\bar{t}^2} \frac{N - n}{Nn}s_e^2 \\ &= \frac{3695^2}{(1067.0 / 36) ^ 2} \frac{129 - 36}{129 \cdot 36} 25.54 \\ &= 7949.7 \end{aligned} \]

For this estimate, the degrees of freedom are 35 (\(n - 1\)), which yields a t statistic of \(t_{\alpha/2}\) = 2.03 (\(\alpha\) = 0.05).

The 95% confidence interval is thus:

\[ \begin{aligned} 95\%\space CI &= \hat Y_k \pm t_{\alpha / 2} \sqrt{var(\hat Y_k)} \\ &= 637 \pm 2.03 \sqrt{7949.7} \\ &= 637 \pm 181 \\ &= [456, 818] \end{aligned} \]

Therefore, based on the 36 videos with a total count of 184 over 1,067 seconds, the estimated total run over this period is 637 +/- 145 fish.

3.2 Total Daily Run

The same calculations were repeated for the second and third periods. The following table summarizes the results of each period.

This figure shows the total estimated run with 95% confidence intervals for each period. As expected, the last period has a higher estimated total than the first period (see the figure of # fish counted per second for the entire day above).

Lastly, the total estimated daily run is calculated by combining the three periods. This table provides a summary.

Therefore, the estimated total daily run on on May 17, 2017 was 23,694 +/- 1,743 fish. This is based on a total of 618 recorded videos that had a total duration of 30,756 seconds (513 minutes). Of the 618 recorded videos, 173 videos were counted yielding a total count of 5,978 fish over 8,215 seconds (137 minutes) with an overall average passage rate of 0.77 fish per second.

For comparison, the total daily run on this day estimated from in-person volunteer counts of 10-minute intervals and using the St2WRS strategy from TR-25 is 24,790 +/- 8,459. And when the video counts during those 10-minute same intervals are used instead of the volunteer counts (but not including the other counted videos), the estimated daily total was 19,115 +/- 4,634.

This table summarizes all three estimates. The totals for the ratio estimator of video counts and St2WRS of volunteer counts are fairly similar; however, the uncertainty band for the volunteer counts is about 5x larger than for the ratio estimator of video counts. This is likely due to larger number of samples used to compute the estimate (i.e. there were more videos across a greater portion of the period), and through the assumption that no fish were passing when videos were not recorded (thus the fish passage rate for the total time without any recorded videos was assumed to be zeo). Lastly, the St2WRS estimate based on video counts had a much lower total than the other two methods likely due a systematic bias between video and volunteer counts (on average, video counts tend to be about 60% of volunteer counts, more on this below).

4.1 Cluster Sampling Method using Video Counts

The total estimated run based on all available video counts using cluster sampling (ratio estimator) method was 410,630.2 +/- 10,822.

4.2 Cluster Sampling Method using Volunteer Counts

To ensure that the cluster sampling method was properly derived, we can use this method to generate daily and total run estimates based on the volunteer count data. These estimates can then be compared to those using the St2WRS method with the same count dataset (see below) to ensure the cluster sampling method produces reasonable results. The two sets of estimates are not expected to be identical because the ratio estimator used in the cluster sampling method is based on the ratio of the total fish to the total observation duration, whereas the St2WRS method is based on the mean of the individual ratios of fish counts to observation time. On some days, volunteers collected observations over longer or shorter periods than 10 minutes, which would cause differences in these two means. However, the results should not be significantly different.

The total estimated run based on volunteer counts using the cluster sampling (ratio estimator) method was 625,554 +/- 60,382.

4.3 Stratified 2-Way Random Sampling using Video Counts

The total estimated run using the 2-way stratified random sampling strategy (St2WRS) and based on video counts was 381,106 +/- 42,859. Note that this only includes videos recorded during the same 10-minute intervals observed by the volunteers, and therefore does not include the entire video count dataset.

4.4 Stratified 2-Way Random Sampling using Volunteer Counts

The total estimated run using the stratified 2-way random sampling strategy (St2WRS) from TR-25 and based on the 10-minute volunteer counts was 628,949 +/- 60,544. Note that this differs from the official DMF estimate (630,098 +/- 60,598) due to differences in the underlying dataset (i.e. DMF used a different version of the volunteer count dataset).

4.5 Comparison of All Four Estimates

This figure shows the total estimated daily run for each of the two methods and each of the two datasets. Error bars are not shown to avoid cluttering the chart. The two volunteer-based estimates generated almost identical results using each of the two methods. The two video-based estimates also yielded comparable results, which were generally lower than the volunteer-based estimates.

This figure shows the cumulative total run over the course of the season using each of the two methods and datasets. The shaded areas represent the cumulative 95% confidence intervals. Again, the two video-based estimates are similar, however, the cluster sampling method produces a much lower uncertainty due to the larger sample size and assumption that fish do not pass when videos are not recorded. The two volunteer-based estimates are very similar in terms of both the total and the uncertainty. Therefore, this sugggests that the cluster sampling methodology was properly derived.

This figure compares the total estimated run with 95% confidence intervals. Again, the two video count-based methods yield similar results, but the Cluster Sampling method has less uncertainty. The two volunteer-based estimates are nearly identical.

Lastly, this table lists the total estimated run and 95% confidence intervals using each method and dataset.

5.1 Number of Periods

By default, each day was split into three 4-hour periods from 7 AM to 7 PM, which was the approach used by DMF in the St2WRS method. In this section, we explore how splitting each day into more or fewer periods affects the estimated totals.

This figure shows the daily estimated run for the number of periods per day between 1 and 12 (1 = 12 hour/period, 12 = 1 hour per period).

This figure shows the estimated total run with 95% confidence intervals. Overall, the results are very similar, although a greater number of periods (6 or 12) led to slightly higher estimated totals.

This table tabulates the estimated total and 95% CI range for each number of periods.

5.2 Minimum/Mean/Median/Maximum Count for Each Video

The video count dataset includes a number of videos that were counted more than once. By default, the mean of those counts were used for each of those videos. However, in many cases there was a wide range in the counts for a single video. Therefore, to evaluate the effect of this variability, the daily and total run estimates were computed using the minimum, mean, median, and maximum count for each video.

This figure shows the estimated total daily run using each statistic of counts among videos counted more than once.

This figure shows the total estimated run with 95% confidence intervals. The results are very similar for the mean and median, but lower using the minimum count per video and higher using the maximum.

The total estimated run and 95% confidence interval range are listed in the following table. The difference in total estimated run between the minimum and maximum of the video counts was 60,592, which is greater than the individual uncertainty ranges. A review of the videos with the largest count variability did not reveal any automated way of identifying which count is more accurate. Therefore, using the mean of multiple counts is recommended for future estimates.

5.3 Number of Counted Videos

To evaluate how many video counts are necessary to obtain an accurate estimate of the total run size, sub-samples of the dataset were generated by randomly selecting a fraction of all existing video counts on each day. For each sub-sample, the cluster sampling methodology was then used to estimate the total run size and its uncertainty. For each fraction, this sub-sampling process was repeated over 100 trials in order to obtain a distribution of estimates for a given fraction of the dataset. The mean values of the estimated total and uncertainty range were then computed over those 100 trials.

This figure shows the estimated total run and 95% confidence interval for each of the 100 trials of each fraction of the dataset. As expected, the variability in the results is higher for smaller dataset fractions.

This figure shows the mean and variability of the estimated total run size for each fraction of the dataset. The variability is represetend using the 5th and 95th percentiles over the 100 trials for each fraction (i.e. this is not the same as the 95% confidence interval of the estimates, which are explored below). The results for the 100% fraction use all available video counts. As expected, as the fraction of the dataset increases, the variability in the estimated total decreases. However, the mean estimated total quickly approaches the total based on 100% of the dataset. With only 1% of the dataset, the estimated total is severely under-estimated.

Lastly, this figure shows the mean and variability (5th and 95th percentiles) of the 95% confidence interval range for the estimated total run size over varying dataset fractions based on the 100 trials. The confidence interval range is calculated as the difference between the upper and lower bounds of the 95% confidence interval of the estimated total (i.e. if the estimated total run is 400,000 +/- 10,000 then the confidence interval range is 2 x 10,000 = 20,000). Using all of the video counts, the CI range was 21,695. But with only 25% of the videos, the mean CI range was about twice as large (49,157). Using only 1% of the video counts, the mean CI range was 158,004.