USER
再翻译：1 Introduction
Visual object tracking targets locating the initialized
target object using a series of bounding boxes
in the color videos. It is one of the most important
tasks in computer vision and provides a good basis
for other vision tasks, such as activity recognition,
person re-identiffcation, and trajectory analysis. Many representative trackers are proposed like MDNet [1],
SiamFC [2], SiamRPN [3], DiMP [4], TransT [5].
Although these Color-based trackers exhibit satisfactory
outcomes in straightforward scenarios, their efffcacy
notably falters under intricate challenging scenarios
like low illumination, over-exposure, fast motion,
heavy occlusion, and background clutter. In these situations,
the RGB camera will generate low-quality
visible color videos while event camera can promote
tracking performance with its bio-inspired sensor and
motion-sensitive character.
To address the limitations arising from singlemodal
defects, some researchers resort to fusing color
and event cameras for reliable object tracking. As the
event camera is a bio-inspired sensor, which records
binary events (i.e., the ON and OFF event) in each
pixel when the variation of light intensity exceeds
the given threshold. It shows signiffcant advantages
over RGB cameras in the high dynamic range (HDR),
low energy consumption, and low latency. The dense
temporal resolution makes it almost free from the
interference of motion blur, and HDR ensures its
imaging performance in low-illumination and overexposure
scenarios. Simultaneously, RGB cameras
provide color and texture information that hold significant
relevance for tracking. The integration of color
and event cameras provides a new avenue for practical
tracking.
Despite their scarcity, a handful of studies have
delved into the fusion scheme for color-event-based
tracking [6–10]. To be speciffc, Zhang et al. [7]
design the cross-domain feature integrator to fuse
the color and event data. Huang et al. [8] reconstruct
grey frames from event streams to enhance the
feature learning for their SVM-based tracker. These
algorithms divide the tracking into multiple stages,
including backbone network, feature fusion, interactive
head, and tracking head, as shown in Fig. 1 (a)-
(b). Note that, Siamese matching based color-event
trackers [6, 7] involve the template and search feature
extraction and matching, which further complicates
the tracking framework. Further, their models achieve
about 20 and 14 FPS only, which makes it hard to
achieve real-time tracking in practical applications.
This inspired us to think about designing a uniffed and
simple multi-modal tracking framework to achieve
efffcient and accurate tracking. The recently proposed
Transformer [11] has shown its natural advantages
and effects in simplifying network structure, such as
object detection [12], segmentation [13], and tracking
[14, 15]. However, there is still no work to design
a uniffed tracking framework for color-event object
tracking task.
In this paper, we propose a uniffed single-stage
tracking framework to accomplish color-event tracking,
termed CEUTrack. As shown in Fig. 1 (c), it
contains three main modules, including the projection
embedding layer, uniffed adaptive backbone, and
tracking head. With the event stream sampled from
DVS, we ffrst reconstruct the event points into voxel
sets for efffcient representation. Accompanied by the
template and search branch inputs, CEUTrack crops
the color frames and event voxel sets to get four regularized
pending tensors. After projection, these inputs
are embedded into multi-modal tokens and fed into
the uniffed vision transformer backbone. To facilitate
information ffow between different Transformer
blocks, we propose the use of adapters to connect
various Transformer layers. Speciffcally, we utilize
cross-attention to aggregate the input and output of
the Transformers, allowing for better capture of richer
multi-level feature representations. The tracking head
projects the output tokens to predict the trajectory of
the target object. Compared with the popular Siamese
frameworks (dual-branch for unimodal, four-branch
for bimodal data), our proposed CEUTrack involves
multi-modal tracking into single-branch architecture
that concatenates all of the tokens as one entry. Therefore,
it simpliffes the multi-modal (color-event) tracking
signiffcantly and achieves a very high running
efffciency (75 FPS). Extensive experiments demonstrate
that our model achieves new SOTA (StateOf-The-Art)
performance on previous and proposed
color-event tracking datasets.
To better validate the effectiveness of our model
and facilitate the community, we also propose a
large-scale benchmark dataset for Color-Event multimodality
Single Object Tracking, termed COESOT. It
contains 1354 aligned video sequences captured by
the DVS346 event camera, in which the training and
testing subsets have 827 and 527 videos respectively.
The target object of COESOT covers a more comprehensive
range of categories than existing color-event
tracking datasets [6, 7], like vehicles, pedestrian, card,
toy, bird, monkey, tiger, elephant, zebra, crocodile, etc.
To fully reffect the vital challenging factors in visual
and event tracking, 17 attributes are annotated to help
detailly evaluate the performance of the trackers. More
detailed introductions and comparisons can be found
in Section 4 and Table 1.
In addition, we also propose a new evaluation
metric named BreakOut Capability score (BOC) for
visual object tracking. Different from existing metrics
that treat each video equally to get a precision
score, such as PR (Precision Rate), NPR (Normalized
Precision Rate), and SR (Success Rate), BOC
score focuses more on the prominence of the evaluated
tracking algorithm compared with existing ones. In
other words, a higher weight proportion will be given
for a challenging video, while an ordinary proportion
will be assigned for a simple video. Note that the difffculty
of each video is comprehensively measured by
the accuracy of existing trackers.
To sum up, the main contributions of this paper
can be summarized as the following four aspects:
• We propose an adaptive uniffed tracking framework
based on a transformer network, termed CEUTrack,
which is the ffrst simpliffed one-stage backbone
for color-event tracking that achieves feature extraction,
fusion, and interactive learning simultaneously.
• We propose a large-scale and general benchmark
dataset for color-event tracking, termed COESOT. It
consists of the most signiffcantly abundant categories
of target objects and video sequences in color-event
tracking community to date. We extend multiple modern
baseline methods on the COESOT for future
works’ benchmark comparison.
• We propose a new evaluation metric for the
tracking task, termed BOC score. It can better reffect
the outstanding ability of the tracking algorithm compared
with the existing baseline methods on difffcult
videos.
• Extensive experiments on multiple benchmarks
(COESOT, VisEvent, FE108) assess the effectiveness
and efffciency of the algorithm. The proposed tracker
CEUTrack sets new SOTA performance on existing
datasets meanwhile running at a remarkable speed (75
FPS).
和 2 Related Work
In this section, we will brieffy review the related
works on event-based tracking and RGB-Event based
tracking. More works can be found in the paper list.
Event-based Tracking. Event-based tracking has
attracted more and more attention in recent years. To
be speciffc, based on the DVS camera, ATSLTD [16]
proposes an event-frame combined algorithm to construct the
event to frames and warp the spatio-temporal
information of asynchronous retinal events. Chen et
al. [17] further feed the sequence of TSLTD frames to
a retinal motion regression network to perform an endto-end
5-degree-of-freedom (5-DoF) object motion
regression. Ramesh et al. [18] use the discriminative
representation for the object with online learning for
long-term event-based tracking. Chamorro et al. [19]
present an ultra-fast tracking algorithm able to estimate
6-DoF motion and a robust mechanism for the
matching of events with projected line segments with
very fast outlier rejection. Ignacio et al. [20] propose
an asynchronous patch-feature tracker that relies
solely on events and processes each event individually
as soon as it gets generated. To solve the HDR situation,
Jiang et al. [21] combined the offfine-trained
detector with an online-trained tracker in a speciffc
Kalman fusion scheme. SiamEvent [22] correlates the
embedded events at two timestamps to compute the
target edge similarity and use the Siamese network
via edge-aware similarity learning. Zhu et al. [23]
proposes an event-based tracker based on key-event
embedding and motion-aware target likelihood prediction.
STNet [24] employs the Transformer network
and spiking neural network (SNN) to extract temporal
and global spatial information. However, with the
above development, event-based tracking still has low
tracking ability because of the limited event-modality
representation and lack of color information.
Color-Event based Tracking. Tracking by combining
color frames and event streams is an interesting
and reliable way to achieve high-performance target
object localization. For instance, Daniel et al. [9, 25]
ffrstly leverage the complementarity of event and visible
cameras to realize low latency feature tracking.
DashNet [26] jointly processes the synchronous activation
from artiffcial neural networks and SNN spikes
signal resources based on the complementary fflter
and attention module. ESVM [8] is a color-event
tracker proposed by Huang et al., which targets tracking
objects by adaptive search region mining and
online SVM (Support Vector Machine) updating. Liu
et al. [27] propose a three-step tracking algorithm by
fusing color frames and event streams simultaneously.
Zhang et al. [7, 28] propose self- and cross-domain
attention with an adaptive weighting scheme to fuse
visual and event images. Wang et al. [6] build the
CMT which is a cross-modality transformer module
that performs well for color-event tracking. Zhang et
al. propose the AFNet [29] which is a dual-branch network
with modality-alignment modules to export the
information fusion of color and event frames to realize
high-frame rate tracking. Although good performance
can be achieved in some scenarios, these trackers
involve multiple components and branches, such as
fusion strategies, multi-step structures, augmentation
modules, and dual-branch (Siamese) frameworks to
locate the target object. Therefore, these algorithms
suffer from heavily limited tracking efffciency with
superffuous module design. In this paper, we propose
a simple and uniffed color-event tracking framework
that can realize more efffcient and accurate tracking.
并分一下intruduction和related work之间的不同。