USER
implement python code for this paper which accepts an foggy image and finds road lane in it "Abstract. In this technological era, the devices are getting more intelligent and smarter with the advent of artificial intelligence and related
technologies. The autonomous vehicle is one of the emerging and important example using the same machine intelligence for effective and
efficient vehicle driving experience. However to make the working of
autonomous vehicle in a smooth and responsive, the accurate lane detection is a crucial aspects and other co-variate like weather condition also
place an important role in it. The present work explores lane detection
in foggy or hazy environment on time series data of a continuous driving scene. The proposed methodology uses Dark Channel Prior (DCP)
to make images un-hazy and this is passed as an input to the proposed
hybrid architecture by connecting a convolutional neural network (CNN)
architecture with a recurrent neural network architecture (RNN). The
model uses Long-Short Term Memory (LSTM) for time-series data analysis as it captures and process the time series data well in advance. In this
model, the CNN block first extracts the feature maps from each frame
which is given as input of the time series data and after this these feature
maps are provided to LSTM to obtain the final prediction. The model
also uses dark channel prior (Image enhancing technique) to enhance
image in case of foggy or hazy environment and then detecting lane with
the help of enhanced images. Finally, the accuracy of the model is shown
in terms of standard performance metrics like precision, recall and F1
score.
Keywords: Deep learning · Convolution neural networks · LSTM · Driver-less driving · Lane detection · Semantic segmentation · Dark
channel prior
1 Introduction
Vehicles were once considered to be the boon of mechanical engineers to the
world. However, there have been many advancements in automobiles in recent
years. Nowadays, the artificial intelligence embed vehicles are getting more attention due to its self-operable behaviours like self-driving cars where no human
c Springer Nature Switzerland AG 2022
I. Woungang et al. (Eds.): ANTIC 2021, CCIS 1534, pp. 596–608, 2022.
https://doi.org/10.1007/978-3-030-96040-7_45
Lane Detection for Autonomous Vehicle in Hazy Environment 597
driver is needed to drive the cars rather the intelligent system of car will automatically drive the car by detecting objects, humans, lanes etc. Keeping the
challenges in view for the autonomous vehicles the detection of lanes accurately
is very important which needs more exploration. It would help the vehicles to
move seamlessly without any collisions or violations of traffic rules. The future
of the automobile industry is to make the vehicles autonomous and provide best
advance driver assistance system (ADAS) possible [1]. Various environmental
factors like fog, dim light etc. are also required to be processed for robust lane
detection.
Lane boundaries detection accuracy has been increased using various algorithms over the years. Introduction to deep learning methods has provided a
huge improvement in the accuracy and robustness. Now using the deep learning
methods with some latest algorithms for image enhancement can improve the
scenario even further. As the lanes are continuous line structures, they can be
either dashed or solid lines. And these scenes are largely overlapped between two
neighbouring frames, so, these neighbouring frames are highly related. Therefore,
lanes in the current frame can be generated by using the information from the
current frame previous frames as well [1].
Meanwhile, deep learning in recent years have proved that they are better in
performance in computer vision problems (i.e. object detection, image classification etc.) and can be relied upon Deep Convulational Neural Networks (DCNN)
which do the work of feature abstraction from images. Deep Recurrent Neural
Networks (DRNN) are used for information prediction for time-series signals. So,
lane detection in continuous frames is a time-series analysis and can be processed
using DRNN. As dimensionality of images can be sometimes quite big enough so
this makes DRNN network slower. That is why DCNN is used as it can reduce
dimensionality keeping intact the important information of the images.
So it can be deduce that there is requirement of hybrid architecture of DCNN
and DRNN for lane detection in continuous frames [1]. Furthermore, applying
this on foggy images might give low accuracy. Hence, it is required to apply Dark
Chanel Prior for foggy or hazy images to remove blurry or hazy effect from the
images before forwarding it to above architecture.
The remaining paper is organized as separate sections in which Sect. 2 is
Literature Review which provides the previous works related to autonomous
vehicles, Sect. 3 is Proposed Model in which the model for lane detection is
explained, Sect. 4 Experimental Outcome and Discussions explains the obtained
result after experiment. Finally, Sect. 5 is Conclusion and Future Scope in which
the entire work is concluded along with the possible works that can be done in
future.
2 Literature Review
The advancement of technologies leads to further extensive research for any
domain. Similarly in the field of autonomous vehicle industry there are numerous
598 B. Kumar et al.
researches has taken place and still going on. In the article [2], the authors have
explored the geometrical methods for the detection of lines, curves etc. in the
figure. Specifically the Hough Transformation method is taken into consideration
for line detection. But it also suffers from some limitations like curved lanes,
sharp turns. These limitations reduces the overall efficiency of the entire system.
The introduction of various artificial intelligence techniques have taken the
problem rectification way at high level. This can be observed in the paper [3]
where a lane detection algorithm is presented which is based on support vector machine (SVM). The SVM pattern recognition is exploited for road surface
extraction. The image is transformed into bird-view image by using world coordinate system (WCS) and image coordinate system (ICS) which providesral point
of the road from the mid-line of the road. The SVM regression is taken into
account for the road shape function. This method is good for dividing a road in
two lanes but fails when there are large number of lanes and we need to detect
all those lanes at same time. Thus this model is not at all suitable for current
needs. In [7], the two group classification problem is also introduced.
The authors in [4] exploited Cascaded CNNs where an end-to-end technique
for lane detection related aspects like boundary detection, classification and clustering is done which is based on two CNNs that runs in real-time. The first CNN
is used to detect lanes using feature extraction techniques. And the second CNN
is used for classifying the type of lanes into 8 categories. This method is not so
robust as it takes into account only one image at a time which may not give
proper lane detection due to weather conditions and obstruction etc.
In the continuation of findings where most of the works has used single frame
for lane detection, the authors in [5] have introduced the similar work by using
continuous frames. In this paper, multiple frames are taken from continuous
scene captured while driving and proposed a architecture which is hybrid in
nature where RNN and CNN is combined. This method although performs good
but still produces bad result when weather conditions are not favourable like
hazy or foggy environment or when sun flare is very strong resulting in high
brightness.
Various approaches is given for the lane detection and in every approach
there are some limitations. Similarly, taking the trade-off related to analysis of
hazy images a Dark Channel Prior algorithm [6] is proposed which has ability to
unhazify an hazy image using single color channel of an image. Using this dark
channel and atmospheric light we can get an unhazy image by interpolation and
soft-matting techniques.
In [8], the concept utilization of a CNN-RNN network is shown. CNN is used
for feature extraction and RNN for doing time series analysis on the features
extracted to make the network robust. Using such concept helps in the action
recognition which is very important for the autonomous vehicles to self operate.
This pipeline of CNN-RNN is useful as it helps to remember useful features and
relations in the data like some sequence or patterns.
Lane Detection for Autonomous Vehicle in Hazy Environment 599
Finding in [9] gives us a brief about Self-driving and driver relaxing vehicle
where the vehicles are focused to be automated to give human driver relaxed
driving. The authors in this work has given two insights related to intelligence
of automated vehicles. The first is automatically following the other vehicle for
being same route, and second is using less resources(breaks, clutches etc.) in
case of heavy traffic. From these insights it can be deduced that how much deep
research enhancement is to be done for an efficient autonomous vehicle system.
A responsive system of automatic driver assistant with functionality of lane
tracking and road signal identification is represented in [10]. Here the important
factors are real time video analysis and optimal utilisation of hardware and
software. The design of proposed system is like it exploits cheaper cameras.
The lane tracking algorithm was implemented on MATLAB in which Hough
Transform was applied.
The paper [11] explores another CNN based approach to the lane detection
problem wherein a lane segmentation network is used to segment out the visible
lane marking pixels before applying a perspective transformation to obtain a
bird’s eye view which facilitates the ease of voting in clustering and curve fitting
that are also applied later. Using this kind of approach provide a base so that
the model can perform satisfactory for accuracy and speed.
The paper [12] explored a vision based lane detection approach which can
perform responsive operations with higher accuracy for the scenarios like light
variation and shadows. The structure of the simulation should be like having
cameras mounted on the vehicle and the images obtained from such camera.
Then the some algorithms are applied for identification of lane or paths.
A deep-learning-based approach for vehicle detection is represented in [13]
that can achieve effective detection performance under extreme conditions. The
technique mentioned in the paper can be useful for in-road driver assistance and
for autonomous driving.
Perception system design or Insight framework configuration is an essential
advance in the improvement of an autonomous vehicle (AV). With the immense
determination of accessible off-the-rack plans and apparently unlimited choices
of sensor frameworks executed in examination and business vehicles, it tends
to be hard to recognize the ideal framework for one’s AV application. The [14]
presents an exhaustive audit of the best in class AV insight innovation accessible
today.
Design of equipment gas pedals for neural networks (NN) applications
includes strolling a tightrope in the midst of the imperatives of low-power, high
exactness and throughput. NVIDIA’s Jetson is a promising stage for implanted
AI which looks to accomplish a harmony between the above destinations. In
this paper [5], an overview of works that assess and enhance neural organization
applications on Jetson stage.
As an essential for independent driving, scene understanding has drawn in
broad examination. With the ascent of the convolutional neural organization
(CNN)- based profound learning strategy, research on scene understanding has
accomplished huge advancement. The paper [15] intends to give an exhaus-
600 B. Kumar et al.
tive review of profound learning-based methodologies for scene understanding
in independent driving.
Lane detection is a critical component in increasing driving safety. In this
research, a lane detection approach for lane departure warning systems that
is both real-time and illumination invariant is presented. The suggested approach [16] works effectively in a variety of lighting circumstances, including bad
weather and at night. Lane detection is an essential part of most ebb and ADASs.
Countless existing outcomes center around the investigation of vision-based path
location techniques because of the broad information foundation and the minimal
expense of camera gadgets. In [13], past vision-based path location studies are
surveyed as far as three viewpoints, which are path identification calculations,
joining, and assessment strategies.
Path discovery in driving scenes is a significant module for independent vehicles and progressed driver help frameworks. Lately, many complex path recognition strategies have been proposed. In any case, most strategies center around
identifying the path from one single picture, and regularly lead to inadmissible execution in dealing with some very terrible circumstances like substantial
shadow, extreme imprint corruption, genuine vehicle impediment, etc. Truth be
told, paths are consistent line structures out and about. Thusly, the path that
can’t be precisely distinguished in one current casing may conceivably be induced
out by joining data of past outlines. To this end, The paper [13] research path
recognition by utilizing different casings of a constant driving scene, and propose a half and half profound engineering by joining the convolutional neural
organization (CNN) and the intermittent neural organization (RNN).
2.1 Research Gap
Traditional methods including geometric modelling and semantic-segmentation
of image works only in the case of single image. This method can under-perform
in cases like serious vehicle occlusion, severe mark degradation, heavy shadow
and so on [1]. Therefore, to advance in this field we need something more concrete
and something we can depend on fully. So we are suggesting a method including
a fusion of DRNN and DCNN to detect lanes. This can detect lanes in continuous
frame.
3 Proposed Model
In order to integrate images taken from consecutive driving scene with previous
images for detecting lanes, we designed a encoder-decoder framework which are
fully convolutional network. For this experiment, we used Tu-Simple dataset
which is a benchmark dataset for road or lane detection. It contains 3700+
sequences of contiguous road images. Each sequence consisting of 20 images
where the 20th image is labeled for ground truth. We changed this dataset to
hazy images using image transformation techniques. We made images hazy and
some brightened images. Then we used this transformed Tu-simple dataset for
Lane Detection for Autonomous Vehicle in Hazy Environment 601
Fig. 1. Architecture of proposed network
this work. Initially. The image is made unhazy by using DCP algorithm. In
large number of non-sky patches there are some one color channel whose pixel’s
intensity is too less and can be said as very near to zero. The dark channel for
an arbitrary image J is given by
Jdark(x) = miny∈Ω(x)(minc∈{r,g,b}Jc
(y)) (1)
The color channel of J is represented by Jx whereas Ω(x) is denoting local patch
which is centered at x. There are two minimum operators which are very important for dark channel minc∈{r,g,b} and miny∈(x). These operators are applied on
each pixel. Along with the minimum filter these operators are also commutative
in nature. Using the concept of a dark channel some facts can be observed like
if there is an outdoor image (excluding any sky region image) J which is hazy
free then the dark channel intensity of image will be very less and tends towards
zero.
Jdark → 0 (2)
With many number of continuous frames of un-hazy images as input the
encoder CNN convert them into time-series feature maps. Convolution and Pooling are used for image abstraction and feature maps extraction.
For the encoder and decoder CNN we are using the pretrained model of
Segnet. It has been successfully trained for semantic segmentation on Imagenet
Dataset. The use of pretrained model provides faster and better training for the
overall model. It can be seen that lane detection is a semantic segmentation
problem and thus we are using encoder-decoder models which will provide an
output having the same size of the input. In the encoder phase image operations
like convolution and pooling are used to extract feature maps from input images.
In the decoder phase deconvolution and unsampling is used to reconstruct the
image and highlight the lanes pixelwise. The pretrained segnet model is presented
in Fig. 2.
602 B. Kumar et al.
Fig. 2. Segnet encoder model
Now, we pass the feature maps extracted from encoder of CNN for time series
analysis. We do this using LSTM (long-short term memory). Modelling the various continuous frames of driving scene as time-series, the RNN block accepts the
feature maps of each frame given by CNN encoder as input. LSTM network has
the ability to forget the unimportant information and remember the essential
ones [1]. Traditional full-connection LSTM is slower as compared to convolutional LSTM. So the convolutional LSTM is used. Convolutional LSTM replaces
matrix multiplication in each gate with convolutional operation. Following are
the activation functions of a general Convolutional LSTM:-
Ct = ft ◦ Ct−1 + it ◦ tanh(Wxc ∗ Xt + Whc ∗ Ht−1 + bc) (3)
ft = σ(Wxf ∗ Xt + Whf ∗ Wt−1 + Wcf ◦ Ct−1 + bf ) (4)
ot = σ(Wxo ∗ Xt + Who ∗ Wt−1 + Wco ◦ Ct−1 + bo) (5)
it = σ(Wxi ∗ Xt + Whi ∗ Wt−1 + Wci ◦ Ct−1 + bi) (6)
Ht = ot ◦ tanh(Ct) (7)
The symbols used in all these equations have some specific role and all these
can be described in specific manner like Xt is the input feature maps at time t
which is extracted by the encoder CNN. Ct, Ht represents memory and output
activation at time t and Ct−1 Ht−1 represents memory and output activation at
time t−1. Ct is cell, it is input, ft is forget and ot represents output gates. Wxi
Lane Detection for Autonomous Vehicle in Hazy Environment 603
denotes the weight matrix of the input Xt to the input gate, bi is the bias of
the input gate. The W and b can be inferred from the already provided rules.
The sigmoidal operation is represented by σ() whereas ‘◦’ and ‘*’ denotes the
Hadamard product and the convolution operation respectively.
Next comes the decoder CNN. In the decoder part, deconvolution and upsampling are used to grasp and highlight the information of targets and spatially
reconstruct them. For effectively decoding the output of LSTM network the size
and number of feature maps should be same as the encoder CNN, while arranged
in inverse direction. Accordingly, the up-sampling and convolution in each subblock of decoder match the corresponding operations in the sub-block of the
encoder [1].
4 Experimental Outcome and Discussions
The proposed algorithm is tested against a TuSimple data-set which consists
3700 sequences of 20th image of each sequence has been labeled which are used
as ground truth in the this work. The labelling contains the (width, height) of
pixels and the belonging lane. So we can easily get the annotated/ground truth
image of the labelled image in each sequence. Time interval between consecutive
images in each sequence is less than 1 s. This lets us use a time series analysis
on the given data-set along with semantic segmentation.
Figure 3 shows a representation of dark channel prior image enhancement
technique which is applied on a input image. The Fig. 3(a) is hazy image while
the Fig. 3(b) is unhazy image which is obtained after applying the proposed
algorithm. It can be clearly seen the clarity of hazy image after applying DCP,
so correspondingly the lane detection model will work much better.
The training of proposed model with DCP is done for 10 number of epochs
and the result is obtained. The same dataset is also tested without using DCP.
The corresponding comparative result for the performance metrices is represented in Table 1.
The average run-time per batch of 100 sequences is 5 s. So, per sequence
average run-time is 50 ms which is better than most other models.
The Fig. 4 shows the plot between accuracy and epochs where it can be clearly
seen that the accuracy increases with number of epoch. The model is able to learn
it’s parameters with more number of epochs. After a certain number of epochs
the model starts over fitting on the data which can be understood at the decrease
in accuracy at last. So the training is stopped as soon as it starts over fitting to
get best bias-variance tradeoff.
604 B. Kumar et al.
(a)
(b)
Fig. 3. Representation of (a) before (b) after, image processed with proposed algorithm.
Table 1. Comparison of performance matrices “With DCP” and “Without DCP”
Measure Score with DCP Score without DCP
Accuracy (in percent) 93.7 91.2
Loss 5.567 × 10−2 8.3469 × 10−2
F1 score 0.6331 0.5999
Precision 0.4937 0.4517
Recall 0.9117 0.8931
Lane Detection for Autonomous Vehicle in Hazy Environment 605
Fig. 4. Accuracy vs epoch plot
Fig. 5. Loss vs epoch plot
The Fig. 5 shows the plot between Loss and Epochs where it can be clearly
observed that the model gets trained with epochs and accuracy increase and
the loss decreases until starts over fitting. When it starts over fitting the loss
increases and training is stopped to get best fit model.
In the Fig. 7 the experiment on n frames vs accuracy is performed. n frames
is the number of contiguous frames used for time series analysis. The experiment
606 B. Kumar et al.
Fig. 6. Application on top of the architecture
Fig. 7. Accuracy vs epoch plot for different n-frames: 5 and 8
on various n frames we choose n = 5 as best option as it gives maximum accuracy
as shown in Fig. 7. With n = 3 and n = 8 we get less accuracy.
The Fig. 6 shows the use of proposed model in a practical continuous road
detection in hazy environment.
5 Conclusion and Future Scope
In this paper, we used a robust hybrid method combining both RNN and CNN
for lane detection in driving scenes. The proposed method includes removing
Lane Detection for Autonomous Vehicle in Hazy Environment 607
haziness from image using dark channel prior method, then using encoderdecoder CNN framework which uses multiple continuous frames as input and
using semantic segmentation predict lanes of current frame. In this we first
extract feature maps of each frame using encoder CNN. Then, the sequential
encoded features of all input frames were processed by a ConvLSTM. Finally,
the outputs of the ConvLSTM were fed into the CNN decoder for information
reconstruction and lane prediction. To get better results in hazy conditions we
also used Dark Channel Prior for image enhancement. The result obtained in
this work is able to support the out-performance of the Dark Channel Prior
algorithm to detect the lanes efficiently for autonomous vehicles operations.
In the future perspective, the integration of other road segments like vehicles,
obstacles and person etc. and create a full fledged ADAS system. Along with
this the enhancement in accuracy in different types of environment and roads
can be taken into consideration."ASSISTANT
Below is the Python code to implement the described lane detection algorithm for foggy images using Dark Channel Prior (DCP) image enhancement and a Convolutional Neural Network (CNN) connected to a Recurrent Neural Network (RNN) with Long-Short Term Memory (LSTM) for lane detection:
```python
import cv2
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers, models, backend as K
from sklearn.model_selection import train_test_split
# Dark Channel Prior for Image Dehazing
def dark_channel(img, size):
min_channel = cv2.min(cv2.min(img[:, :, 0], img[:, :, 1]), img[:, :, 2])
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (size, size))
dark_channel = cv2.erode(min_channel, kernel)
return dark_channel
def atmospheric_light(img, dark_channel):
h, w = img.shape[:2]
num_pixels = h * w
num_brightest = int(max(num_pixels / 1000, 1))
dark_vec = dark_channel.ravel()
img_vec = img.reshape(num_pixels, 3)
indices = dark_vec.argsort()[num_pixels - num_brightest::]
atmsum = np.zeros((1, 3))
for ind in indices:
atmsum += img_vec[ind]
A = atmsum / num_brightest
return A
def dehaze(img, omega=0.95, t_min=0.1, window_size=15):
dark_channel_img = dark_channel(img, window_size)
A = atmospheric_light(img, dark_channel_img)
transmission = 1 - omega * dark_channel_img / A
transmission = cv2.max(transmission, t_min)
transmission = cv2.merge([transmission, transmission, transmission])
img_dehazed = (img - A) / transmission + A
img_dehazed = cv2.max(cv2.min(img_dehazed, 255), 0).astype(np.uint8)
return img_dehazed
# CNN-LSTM Model for Lane Detection
def build_model(input_shape):
# CNN Encoder
cnn_input = layers.Input(shape=input_shape)
x = layers.Conv2D(32, (3, 3), activation='relu', padding='same')(cnn_input)
x = layers.MaxPooling2D((2, 2))(x)
x = layers.Conv2D(64, (3, 3), activation='relu', padding='same')(x)
x = layers.MaxPooling2D((2, 2))(x)
cnn_output = layers.Conv2D(128, (3, 3), activation='relu', padding='same')(x)
# LSTM for sequential data
lstm_input = layers.TimeDistributed(cnn_output)(cnn_input)
lstm_output = layers.ConvLSTM2D(64, (3, 3), padding='same')(lstm_input)
# CNN Decoder
x = layers.Conv2DTranspose(64, (3, 3), activation='relu', padding='same')(lstm_output)
x = layers.UpSampling2D((2, 2))(x)
x = layers.Conv2DTranspose(32, (3, 3), activation='relu', padding='same')(x)
x = layers.UpSampling2D((2, 2))(x)
x = layers.Conv2D(1, (1, 1), activation='sigmoid')(x) # Output layer for lane detection
model = models.Model(cnn_input, x)
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
return model
if __name__ == "__main__":
# Load the dataset (Simulated using random data for demonstration purposes)
# Assuming `frames` is a list of contiguous frames and `labels` is the corresponding ground truth
frames = np.random.rand(100, 20, 256, 256, 3) # Example data
labels = np.random.rand(100, 256, 256, 1) # Example labels
# Process images with Dark Channel Prior (DCP)
frames_dcp = np.array([[dehaze(img) for img in sequence] for sequence in frames])
# Split data for training and testing
(train_frames, test_frames, train_labels, test_labels) = train_test_split(frames_dcp, labels, test_size=0.2)
# Build and train the model
input_shape = (20, 256, 256, 3) # Example shape
model = build_model(input_shape)
model.summary()
# Training the model
history = model.fit(train_frames, train_labels, epochs=10, batch_size=1, validation_split=0.2)
# Evaluate the model
test_loss, test_accuracy = model.evaluate(test_frames, test_labels)
print(f"Test Accuracy: {test_accuracy*100:.2f}%")
```
### Key Points:
1. **Dark Channel Prior (DCP)**: This part deals with image enhancement. The `dark_channel`, `atmospheric_light`, and `dehaze` functions implement the dehazing technique.
2. **CNN-LSTM Model**: The `build_model` function constructs the CNN-LSTM architecture for lane detection.
3. **Training Data**: Simulated random data has been used to demonstrate how to fit the model. You should replace it with your actual dataset, ensuring it is pre-processed into contiguous sequences.
4. **Evaluation**: After training, the model is evaluated on a test set, and performance metrics are printed.
### Note:
- For actual deployment, the dataset loading and pre-processing need to handle real driving scenes and ground truth data.
- The code assumes the TuSimple dataset or equivalent is transformed beforehand. The example uses random data solely for demonstration. Substitute the placeholder code with actual data handling routines.