USER
You are an expert in Deep Learning and Machine Learning. As an Academic Peer Review Assistant, your role is to provide a thorough and constructive review of submitted academic papers. When a user presents a paper for review, assess it across several key dimensions to ensure scholarly rigor and clarity. The paper has submitted to ECIR (european conference on information retrieval) reproducibility track. Your evaluation should cover:
• Content Accuracy and Relevance: Analyze whether the information presented is accurate, up-to-date, and relevant to the stated research questions.
• Argumentation and Analysis: Assess the strength and coherence of the arguments. Are the claims well-supported by evidence? Is the analysis thorough and critical?
• Methodology: Evaluate the appropriateness and execution of the research methodology. Is it suitable for the research question? Are the methods clearly explained and justified?
• Organization and Structure: Consider the logical flow of the paper. Is it well-organized, allowing readers to easily follow the progression of ideas?
• Writing Quality and Clarity: Review the paper for clarity of language, style, and grammar. Is the writing clear, concise, and appropriately academic?
• References and Citations: Check for the completeness and accuracy of citations and references. Are all sources credible and properly attributed?
Provide specific feedback for each category, citing examples from the paper to support your evaluations. Offer constructive criticism and suggestions for improvement to help the author enhance the paper. Summarize the overall strengths and weaknesses, and recommend whether the paper should be accepted, revised, or rejected based on your assessment.
please format the review in those following sections, explaining in great details all aspects you have thought until now.
the sections to reformat the review into are:
Paper Summary
strengths
limitations
Overall evaluation
this is the paper:
# An investigation into the inconsistency of shot
# boundaries and evaluation protocols within video
# summarization
```
Anonymous Authors
```
```
Abstract.Video summarization is an important application within com-
puter vision and video retrieval, producing concise summaries. Deep
learning-based video summarization relies on two critical steps: first di-
viding the video into disjointed segments, referred to as "shot bound-
aries" and second the rank correlation based evaluation protocol to assess
model performance against human annotations on benchmark datasets.
These steps are crucial as appropriate shot boundaries and strong cor-
relation metrics significantly influence the perceived quality of the video
summary. However, the details underlying both procedures are ambigu-
ous, raising concerns regarding the replicability of the shot boundaries
and reproducibility of model performance outcomes in video summa-
rization. This work describes a replication and reproduction study of the
shot boundary detection proposed by previous research. Furthermore, we
conducted a reproduction study of various evaluation protocols described
by prior work. Our work failed to replicate the shot boundaries provided
by previous research and the reproduced shot boundaries exhibit incon-
sistencies across different setups. We also demonstrate that the appli-
cation of different post-processing steps can yield substantial variations
in the rank correlation coefficients, suggesting that post-processing steps
may significantly skew evaluation outcomes. Based on these results, we
recommend that future benchmark datasets introduce a dedicated shot
detection sub-task alongside expert annotated shot boundaries to en-
sure summary quality. We also advise against the use of post-processing
before evaluation, advocating instead for the direct use of model predic-
tions. Finally, we call for more rigorous documentation of post-processing
procedures used in evaluation.
```
```
Keywords:Video summarization·Replication Study·Methodology
Evaluation.
```
## 1 Introduction
Video summarization is a critical application within video retrieval systems and
a important task in computer vision. Its primary goal is to identify and extract
the most salient segments of a video to create a concise summary that captures
the video’s essence [19]. This task plays a central role in retrieval and search,
by facilitating the curation of video databases [1] and enabling the creation of
personalised summaries [18, 20]. The predominant approach to summarize videos
2 A et al.
employs deep learning models [1], with many of these models benchmarking their
performance on the widely-used TVSum [23] and SumMe [8] datasets. These
datasets are favored due to their diverse video content and multiple human
annotation per video. The video summarization approaches can be broken down
into four key steps (depicted in Figure 1), three of which are relevant to our study.
The first step involves pre-processing the video through shot boundary detection,
where the video is segmented into disjointed sections that will later form the
summarized video [21]. The second step, based on the framework proposed by
Zhang et al [28], includes a training and inference pipeline to learn each frame
relevance, denoted as “ importance score” to the final summary. These scores
alongside the shot boundaries inform the third step, which utilises aknapsack
solver to rank the shots with the fewest frames, but the highest importance
scores to construct a summary within a predefined length constraint.
While these steps have been key for many developments in video summariza-
tion, it has not gone without scrutiny. Otani et al [16] noted a bias introduced
by the shot-boundary detection and the knapsack solver which allows random
scores to achieve state-of-the-art performance using the widely-used F1 metric.
The authors proposed instead to use the Kendall/Spearman Rank Correlation
coefficient to avoid this bias. They also note the importance of the shot bound-
aries as it plays a central role in the perceived summary quality alongside rank
coefficients.
Given these considerations, it is essential to provide a clear procedure for
generating shot boundaries. While several works have used the shot boundaries
provided by Zhang et al [28] for the benchmark datasets TVSum and SumMe,
the procedure followed to create the shot boundaries using the “Kernel Tem-
poral Segmentation" (KTS) algorithm is not provided in detail. This lack of
transparency raises two concerns: first, whether the provided boundaries are
replicable, and second, whether any changes made to KTS such as different in-
put features dramatically alter the results. The latter issue is especially pertinent
as prior research [7, 17, 25] in video summarization has explored training models
using different feature representations. It is reasonable to assume that features
used in model training could also be used with KTS. If such modifications lead
to inconsistent results, this variability could in turn impact the reproduciblity
of video summaries across different systems. Therefore we introduce our first re-
search question:RQ1Can we replicate the shot boundaries provided by previous
research? If we reproduce the shot boundaries using different feature represen-
tations and alter this process, are those shot boundaries consistent with each
other?
Alongside this, it is also important to examine Otani et al. [16] evaluation ap-
proach. We observe the processing applied to the models prediction/annotations
prior to estimating the rank correlation coefficients. We also note that they do
not report results on the SumMe dataset. This raises concerns as to whether
different video summarization methods consistently measure the correlation in
the same way. This concern is especially pertinent as several works report re-
sults on both the TVSum and SumMe dataset [5, 10, 22, 27], but only a few of
```
Title Suppressed Due to Excessive Length 3
```
Fig. 1: Overview of the video summarization pipeline, consisting of 4 steps: (1)
frame-wise pre-processing, (2) shot-boundaries detection using the KTS algo-
rithm, (3) frame-wise score prediction using a machine learning model and (4)
knapsack based post-processing to generate the final summary.
them provide the code [10, 22] and/or a description of their evaluation proce-
dure [5]. This renders model comparison and reproduction difficult as the type
of applied post-processing could dramatically affect the results. Therefore we in-
troduce our second research question for our work:RQ2Given the same model,
to what extent do different post-processing steps followed prior to evaluation
lead to different results in terms of Kendall correlation coefficients?
We addressRQ1by conducting a study to replicate the original shot bound-
aries under identical conditions and to reproduce them under different setups.
Initially, we follow Popatov et al [21] procedure to replicate the shot boundaries
as was mentioned by Zhang et al [28]. We then alter their procedure to reproduce
those shot boundary under different settings, by utilizing features extracted by
a Convolutional Neural Network (CNN). This choice was made as such features
have been used in training Deep Neural Networks applied in video summariza-
tion, so it would be natural to assume that they could be applied in this context
as well.
We investigateRQ2by reproducing the training and evaluation procedure
followed by previous works [1, 3, 5–7, 22] under four post-processing scenarios
inspired by recent research [3, 5, 7, 22] prior to inference. We experiment with
sub-sampling/up-sampling and the use of the knapsack algorithm and demon-
strate the differences in the Kendall correlation coefficients when different post-
processing is applied to the model’s output.
For each of these experiments, we provide the code in our repository^1. Our
analysis further reveals that certain strategies may introduce existing biases in
evaluation, which we detail in Sections 5 and 6.
(^1) https://anonymous.4open.science/api/repo/VidSumMethods-565B/zip
4 A et al.
## 2 Related works
In this section, we first briefly describe relevant datasets and popular approaches
to video summarization. We then highlight previous research which has studied
methodological flaws in these approaches.
Video SummarizationVideo summarization [1, 19] is the task of retrieving
the most relevant frames of a video to construct a summary. For this work,
summaries are generated in the form of a video skim. Several video summa-
rization approaches have centered around the nature of the data, and datasets
such as TVSum [23], MED-Summaries [21] and SumMe [8] were introduced to
tackle the task of “Generic" video summarization. The aforementioned works
also contributed the Shot Boundary detection algorithm; Kernel Temporal Seg-
mentation and the creation of a post-processing pipeline with a knapsack solver
to create summaries. The approach to model video summarization is driven by
deep learning, which was marked by the contribution of an LSTM based model
[28]. These approaches [1] largely followed the same pre-processing and evalu-
ation procedure, with different approaches in the model architectures [6, 26],
training strategy [14] or feature extractors [7, 13].
Methodological challenges in video summarizationSome works have ex-
plored methodological issues in popular approaches. Otani et al [16] demon-
strated that the pre and post processing steps followed in the video summa-
rization pipeline bias the model’s predictions based on the shot boundaries.
Apostolidis et al [2] showcased that model comparison based on benchmark per-
formance was difficult due to varying difficulties over different cross validation
configurations. Alongside these issues, some papers have noted methodological
concerns when examining prior research [7, 25], reporting challenges such as
overlapping test splits. In contrast to these works, we specifically investigate the
replicability of the shot boundaries provided by previous research and examine
the consistency of the evaluation protocol proposed by Otani et al [16] and used
by various works [1, 3, 5–7, 22].
## 3 Methodology
In this section, we outline the methodology for our replication and subsequent
reproduction study in case ofRQ1and reproduction study forRQ2. We broadly
follow the pipeline employed by previous works [6, 28], while we provide a de-
tailed description of the used software and hardware in our repository.
3.1 Video Summarization Pipeline
The video summarization pipeline follows four main steps, as seen in Figure 1: (1)
frame-wise feature extraction, (2) shot boundary detection via Kernel Temporal
Segmentation (KTS), (3) model training and inference, and (4) creation of the
video summary by knapsack post-processing.
```
Title Suppressed Due to Excessive Length 5
```
(1) Pre-ProcessingConsider a videoVwhich is comprised ofs×Nframes. Let
the video be sub-sampled by a factorsto an ordered setVs= {F 1 ,F 2 ,F 3 ..FN}
composed ofNframes. This ordered set is passed to feature extractorEper
element, which results inE(Vs) = {Fˆ 1 ,Fˆ 2 ,Fˆ 3 ..FˆN}whereFˆi=E(Fi),i∈N.
The feature extractorE could be represented using a Fisher Vector feature
extractor [15], or a Convolutional Neural Network [11, 12, 24].
(2) Kernel Temporal Segmentation Described by Popatov et al [21], the
KTS algorithm estimates the shot boundaries, called in this context as "change-
points" of the signalX = x 1 ,x 2 ,x 3 ,x 4 ,..xn, wherexdenotes the extracted
features. The algorithm aims to minimise the cost functionJ(m,n), using two
optimization criteria balanced by a regularization parameterC, wheremrepre-
sents the number of change points andnis the length of the signal. The first
criteria minimizes the variance within change points as a loss functionL(m,n)
and the second penalises the creation of too many change points, being repre-
sented byg(m,n). The dual objective loss function is provided in equation 1.
The temporal parametrization of the signal is done via a Linear Kernel function
K:X×X→R, whereφ(xT)denotes the value of the kernel function at time
T.
Minimizem;0, 1 , 2 ...,m− 1 Jm,n:=Lm,n+Cg(m,n) (1)
whereLm,nis defined as follows:
```
Lm,n=
```
```
Xm
```
```
i=
```
```
vtT− 1 ,Ti, vTi,Ti+1=
```
### TX+
```
Ti
```
```
||φ(xT)−μi||^2 μi=
```
### P
```
Tit+1φ(xT)
Ti+1−Ti
```
### (2)
The above cost function is optimized via a Dynamic programming solver with
backtracking, and the implementation is provided here^2.
(3) Model TrainingVideo summarization is formulated as a regression task
for the machine learning model. Consider a sequenceS={Fˆ 1 ,Fˆ 2 ,Fˆ 3 ..FˆN}where
Fˆi, i∈Ndenotes the extracted features from the video frame at index i. This
sequence is then provided to modelMwhich estimates the "importance", leading
to sequenceM(S) ={I 1 ,I 2 ,I 3 ...IN}whereIidenotes the prediction at indexi.
The model is optimized with ground truth annotated scoresIt={Iˆ 1 ,Iˆ 2 ,Iˆ 3 ...IˆN}
using the mean squared error:errS=MSE(M(s),It).
(4) Knapsack Post-ProcessingConsider the predicted sequence :M(S) =
{I 1 ,I 2 ,I 3 ...IN}, and a set ofm+ 1shots produced by KTS:
SH ={SH 1 ,SH 2 ,...,SHm+1}, where each shot SHj,j ∈m+ 1contains
a subset ofljframes{Fj,...,Fj+lj}from videoVs. Next, the corresponding
sequence of importance scores{Ij,...,Ij+lj}is up-sampled to the original frame-
rates×Nresulting into{Ij,...,Ij+lsj}, where the length of the up-sampled
(^2) https://github.com/wulfebw/algorithms/blob/master/scripts/dynamic_programming/knapsack.py
6 A et al.
sequence islsj. Then, an average importance score is computed for all frame
indices between[j,j+lsj], represented assimpj=
Pj+lsj
j Ij
s×lj. This results in a
set of shot scoresSHimp={simp 1 ,simp 2 ,...,simpm+1}. This set is then given
to the knapsack solver which selects the shots which provide the highest score
with the shortest length based on a budget, typically set to15%of the video’s
length ( 0. 15 ×s×N). Finally, a summary is generated by returning a subset
ofkshots{simpz},z∈kwhich possess the largest average importance with the
fewest frames (denoted as 1s in Figure 1).
```
(a) Scenario 1 (b) Scenario 2
```
```
(c) Scenario 3 (d) Scenario 4
```
Fig. 2: A schematic representation of different evaluation scenarios in the post-
processing phase, based on computing the Kendall correlation coefficients.
## 4 Experimental Design
For our set of experiments we first describe the employed datasets, followed by
the replication and reproduction experiments for answeringRQ1in Section 4.
and finally we describe the reproduction experiments designed for addressing
RQ2in Section 4.3.
```
Title Suppressed Due to Excessive Length 7
```
```
Table 1: Description of dataset characteristics, considered in this study.
DatasetDuration(mins)Videos Topics Annotation Style
TVSum 3-10 50 news, how-to’s,documentaries 1 - 5 rating
SumMe 1-6 25 holidays, events, sports 0/
```
4.1 Datasets
We used the TVSum [23] and SumMe [8] datasets in our experiments. In par-
ticular, we utilise both the original provided videos as well as the pre-processed
dataset provided by Zhang et al [28]. Both datasets provide multiple human
annotations per video, which serve as ground truth for evaluating our model’s
summarization performance. However, each dataset provides a different style of
annotation, as displayed in Table 1.
4.2 Study of Shot Boundary Detection
This set of experiments initially attempts to replicate the shot boundaries pro-
vided by Zhang et al [28] which used Popatov et al [21] procedure and then
reproduces the shot boundaries under different setups inspired by Zhu et al [30]
code. We use theopencv2python package to read the videos. These setups differ
mainly in the feature extraction and sub-sampling rate. The procedures for both
are detailed below.
We describe the procedure proposed by Popatov et al [21] as follows. First,
the video is sub-sampled by selecting every 5th frame. Next, we use the skimage
version 0.24.0 to extract the SIFT features per frame and then apply PCA using
sklearn.decomposition.PCAto extract 64 features after dimensionality reduc-
tion. For each of these sub-sampled features, we useskimage.feature.learn_gmm
to train a Gaussian Mixture Model with 128 components and finally extract
the features vectors [15] usingskimage.feature.fisher_vector, resulting in a
D = 16512vector for each frame.
For our reproduction experiment, we adapt the procedure highlighted in the
code provided by Zhu et al [30]. For every frame from the video, we extract the
features by passing each frame through a Convolutional Neural Network (CNN)
and retrieve the features from the penultimate layer from each network. We use
the implementation^3 of three CNNs: GoogleNet [24], ResNet [11] and DenseNet
[12] initialised with their ImageNet weights. These models were chosen as each
of them are popular feature extractors for computer vision tasks and/or have
been used previously for video summarization [7, 17, 22].
Both of these pre-processing steps provides us with a sequence:
S={Fˆ 1 ,Fˆ 2 ,...,FˆN}whereFˆi, i∈Nrepresents the extracted features
at frame indexi. This sequence is provided to the KTS algorithm to create
shot boundaries. These shot boundaries are evaluated against the original shot
boundaries provided by Zhang et al [28] using the F1 metric which has been used
(^3) torchvision==0.14.0+cu
8 A et al.
to compare shot boundaries in previous research [9]. We also examine whether
shot boundaries extracted by CNN based feature extractors are consistent with
each other. Our goal is to establish whether the potential variations between
extracted features could lead to discrepancies in shot boundary detection.
We perform this experiment for the shortest 10 videos from the TVSum and
SumMe dataset. This choice was done as the algorithms time complexity scales
to O(mn^2 ) (with parametersm,nintroduced in Section 3.1). Finally, we conduct
an ablation study for finding the best parameterC(introduced in Section 3.1).
4.3 Post-Processing using Kendall Correlation Coefficient
For this study, we examine the effect different post-processing procedures may
have on estimating the Kendall coefficient. For this, we adopt the training pro-
cedure described by previous works [3, 6, 7, 25], while simulating four inference
scenarios. The training procedure is a five-fold cross validation split using all
videos from each dataset^4. The overall performance for each scenario is reported
as the average over each split’s highest correlation, we report this after five iter-
ations of the procedure. The model implemented for this work is the Multi Layer
Perceptron^5. Next, we describe each scenario, while their schematic representa-
tion can be seen in Figure 2.
Scenario 1 For each video, we sub-sample the annotation scores to match the
size of the model’s prediction and we estimate the correlation. The TVSum an-
notations are scaled from 1-5 to 0-1 and the final correlation is reported as the
average correlation between each human annotator and the model’s prediction.
The SumMe dataset provides a binary annotation, therefore, we compute the av-
erage over all user annotations (15-20 annotators per video), scaling the obtained
values to 0-1 which are then used to estimate the correlation. This procedure was
adapted from the code provided by [3, 4] which only applies it to the TVSum
dataset, while we apply it to both benchmark datasets.
Scenario 2We apply the knapsack post-processing to the model’s prediction and
then estimate the Kendall correlation coefficient with respect to the annotations.
This results in a binary prediction with the same length as the ground truth
annotations. The annotations are processed in line with Scenario 1. This scenario
was adapted from the code provided by [7, 22]^6. The knapsack solver in this case
is given a budget of15%length of the original video.
Scenario 3We up-sample the model’s prediction to match the sample rate of the
annotations and then estimate the correlation between them. The annotations
are processed in line with Scenario 1. This scenario was independently intro-
duced in this work to measure the difference between up-sampling the model’s
prediction versus sub-sampling the ground truth.
(^4) 50 in TVSum, 25 in SumMe
(^5) using Pytorch version 1.
(^6) [7] Compares the knapsack post-processed annotations with that of the knapsack
processed model predictions, but this scenario is not explored within this work
```
Title Suppressed Due to Excessive Length 9
```
Scenario 4For this scenario, the sub-sampled ground truth used to optimize the
model is correlated with the predictions produced by the model. This ground
truth is created as a normalized average annotation score in the TVSum and
SumMe datasets. Note: in the case of the SumMe dataset, Scenario 4 and Sce-
nario 1 are the same. This scenario was examined as it was previously proposed
by [5].
## 5 Results
We begin by presenting the findings related toRQ1, which focuses on the replica-
tion of shot boundaries. This is followed by an analysis ofRQ2, which examines
the impact of various post-processing scenarios and evaluates results using the
Kendall correlation coefficient.
5.1 Shot boundaries Replication and Consistency
Initially, we expected that the replication experiment will retrieve the same shot
boundaries within a margin of error and that the shot boundaries between dif-
ferent feature extractors remain consistent. Our study however resulted in a
failure to replicate the shot boundaries provided by previous research [28] and
the reproduction study demonstrated a significant inconsistency under different
settings. The results illustrated in Table 2 demonstrate that in all cases, the shot-
boundaries differed significantly from the ones provided by [28]. The variations
of theCparameter (introduced in Section 3.1) showed that reducing the penalty
in creating more shot boundaries also reduced the F1 score. In the case of the
Fisher Vector and DenseNet feature representations, the algorithm returns the
same shot boundaries despite changes in theCparameter, resulting in the same
score. Moreover, the highest F1 score is achieved using the GoogleNet features,
but this is still quite distant from the original one. The consistency between
Table 2: The F1 scores between the reproduced vs. the provided shot boundaries
from Zhang et al [28] (higher is better).
Feature Extractor C=1.0 C=0.8 C=0.6 C=0.
TVSum SumMeTVSum SumMeTVSum SumMeTVSum SumMe
Fisher Vector 0.029 0.024 0.031 0.029 0.031 0.029 0.031 0.
GoogleNet 0.451 0.137 0.393 0.102 0.326 0.092 0.237 0.
ResNet 0.359 0.110 0.316 0.090 0.263 0.077 0.197 0.
DenseNet 0.082 0.121 0.082 0.121 0.082 0.121 0.082 0.
the generated shot boundaries is also quite limited. As seen in Table 3, the shot
boundaries generated by different feature extractors disagree with each other
significantly, with a higher disagreement in the case of the SumMe dataset.
10 A et al.
Table 3: F1 score measuring the consistency between shot boundaries created
using the CNN feature extractors (GoogleNet, ResNet and DenseNet), for various
values of the C parameter (higher is more consistent).
CTVSumSumMe
1.0 0.271 0.
0.8 0.252 0.
0.6 0.231 0.
0.4 0.212 0.
Table 4: The Kendall Coefficient across different post-processing scenarios, where
GT denotes the ground truth annotations (higher is better).
ScenarioSub-sample GTUp-sample PredictionAverage GTKnapsack Correlation
TvSum SumMe
1
√
× × × 0.174 -
2 ×
√
×
√
0.100 0.
3 ×
```
√
× × 0.173 0.
4
```
```
√
×
```
```
√
× 0.307 0.
```
5.2 Post-Processing Scenarios Effect on Kendall Coefficient
On examination of the results over each scenario described in Section 4.3 and
illustrated in Table 4, we note significant deviations in the reported correlation
scores across each scenario. In particular, the use of the post knapsack processed
predictions as described in Scenario 2 results in a score 2. 1 times that of Scenario
1 ( 0. 072 vs 0. 154 ), which does not modify the model’s predictions in case of
the SumMe dataset. Furthermore, in the case of the TVSum dataset, there is
a notable difference between Scenario 1 and Scenario 4, which computes the
correlation with the optimized ground truth leading to a score of 0. 307. The latter
shows that ground truth-based annotations yield a higher correlation coefficient
than the model’s predictions in the case of the TVSum dataset^7.
When we compare our model performance in different scenarios with respect
to the state-of-the-art, it becomes apparent that depending on the evaluation
scenario, our model can be quite distant from the state-of-the-art, or come within
60% to it (0.154 (ours) vs 0.246 [22]). This observation further highlights the need
for clarity in the post-processing applied prior to evaluation.
## 6 Discussions
The reported results underscore two main challenges: the first being in the repli-
cation of shot boundaries provided by previous research and the second lying in
the difficulty in comparing models on the benchmark datasets due to differences
in the post-processing stage.
(^7) Note that Scenario 4 is left blank in the case of the SumMe dataset as for this
dataset, both Scenario 1 and Scenario 4 follow the same procedure.
```
Title Suppressed Due to Excessive Length 11
```
Table 5: Comparison of the reported Kendall Correlation Coefficients alongside
the results of our study.Bold textin the Model column indicates the evaluation
scenario is described in the paper, while the Code description column indicates
that the evaluation is only available in the code.
Model Scenario TypeCode description Split typeTVSum SumMe
Existing Work
A2Summ [10] 1,
√
1 ×5 FCV 0.137 0.
MAAM[25] Unknown - 1 ×5 FCV 0.207 0.
Clip-It [17] Unknown - 1 ×5 FRV 0.108 -
SumGraph [29] Unknown - 1 ×5 FCV 0.094 -
PGL-SUM [3] 1,-
√
1 ×5 FRV 0.150 -
MSVA [7] 1,2 - 1 ×5 FRV 0.190 0.
CSTA [22] 1,
√
1 ×5 FCV 0.194 0.
VideoSage[5] 4,
√
1 ×5 FCV 0.300 0.
Baselines
MLP 1,1 - 1 ×5 FCV 0.174 0.
MLP 2,2 - 1 ×5 FCV 0.100 0.
MLP 3,3 - 1 ×5 FCV 0.173 0.
MLP 4,4 - 1 ×5 FCV 0.307 0.
6.1 Implications of shot boundaries investigation
Considering that our replication study resulted in a failure to retrieve the orig-
inal boundaries and the boundaries themselves are not consistent with each
other, this highlights a key challenge in video summarization research. Otani et
al [16] have demonstrated that accurate shot boundaries are essential for pro-
ducing high-quality video summaries. Consequently, it is important that these
boundaries are tailored to the unique content of each video. When boundaries
are inconsistent, even a model capable of effectively ranking important content
may fail to generate high-quality summaries. Determining the optimal approach
to shot boundary detection remains challenging, as different videos frequently
require distinct boundary delineations. Therefore, creating a dataset encompass-
ing a wide variety of video types, using expert annotations for shot boundaries
and user studies with baseline models to validate the quality of generated sum-
maries, would be a valuable advancement. Alongside this, in order to ensure the
robustness of the shot boundary detectors, we recommend incorporating shot
boundary detection as a dedicated sub-task within video summarization.
6.2 Kendall Correlation and Random Scoring
In the challenge of benchmark comparison, utilizing the knapsack processed pre-
diction also reintroduces a problem highlighted by Otani et al [16], that random
scoring can still result in high F1 scores. We conduct another experiment where
we simulate the training procedure, but we instead generate 200 random strictly
positive predictions sampled from a normal distribution. We compute the average
12 A et al.
correlation of the random prediction with ground truth annotations and record
the best obtained correlation sampled from the random prediction. We perform
this for two scenarios: Scenario 1, which does not process the predictions and
Scenario 2 where the knapsack post-processing is applied to the random scores.
We report the scores over the entire SumMe dataset in Table 6, which demon-
strates that the knapsack post-processing still introduces a bias, even when under
the Kendall correlation protocol. As expected, the average random results in no
Table 6: An illustration of the boosted performance introduced by the post
Knapsack processing stage.
ScenarioAverage CorrelationBest Correlation
1 0.000 0.
2 0.096 0.
correlation, but the use of the knapsack introduces a bias which dramatically
improves the performance. In addition, if we select the best random prediction,
simulating the selection of the best model from a validation split also beats
the state-of-the-art performance (0.277 vs 0.246 [22]). This highlights why the
knapsack post-processing must not be used during the model evaluation.
## 7 Conclusions
This work explored two important steps in video summarization: first, the repli-
cability and reproducibility of shot boundaries generated by the Kernel Tempo-
ral Segmentation algorithm, and second, the impact of different post-processing
strategies on the correlation-based evaluation protocol. We were unable to repli-
cate the shot boundaries based on the information in the original methodol-
ogy, and the shot boundaries showcased inconsistencies when reproduced using
different feature extractors. Additionally, we highlight the difficulty in compar-
ing methods as the reported Kendall coefficient diverge when different post-
processing techniques, particularly knapsack post-processing, are applied. To
address the shot boundary replication problem, future work should introduce
datasets with expert annotated shot boundaries and conduct user studies to
ensure the metrics accurately reflect summary quality. We also recommend in-
corporating shot boundary detection as a benchmark task within summarization.
Additionally, we strongly recommend that studies using the TVSum and SumMe
datasets illustrate post-processing steps before evaluating correlations. To ensure
more reliable results with Kendall correlation, studies should avoid using knap-
sack post-processing and instead evaluate the model predictions directly. Finally,
we provide our code to ensure the reproducibility of this work.