A Streaming Approach to Data Discrepancy Detection and Adaptation in Deep Neural Networks

Chambers, Lorraine (2024) A Streaming Approach to Data Discrepancy Detection and Adaptation in Deep Neural Networks. Doctoral thesis, Birmingham City University.

Preview

Text Lorraine Chambers PhD Thesis published_Final version_Submitted Dec 2023_Final Award Jun 2024.pdf - Accepted Version Download (2MB)

Abstract

Deep learning has achieved remarkable success over the last ten years. However, keeping Deep Neural Networks (DNNs) up to date with changing data remains at the forefront of creating genuinely practical systems. To address some aspects of this challenge, this thesis studies the detection of streaming changing data and the subsequent adaptation of DNNs. The main challenges are to efficiently detect the changes and update the DNN promptly without forgetting the pertinent older information. This presents more of a challenge for DNNs than for other traditional machine learning models as DNNs take longer to train, require more data and suffer from catastrophic forgetting where previously learnt classes are forgotten when the DNN is adapted to the new data. DNNs are typically used with high dimensional unstructured data as opposed to lower dimensional structured data, which the streaming literature has focused upon thus far. Hence, DNN adaptation has not been widely studied in the streaming machine learning literature.

So far, clustering is the preferred method for detecting changes in data however, this can be slow as a number of instances must be received before a change is detected. A DNN is usually considered as a ’black box’ where, given an input, it provides outputs, but the specific process by which it arrived at the output is not easily discernible. Inside this ’black box’ are many artificial neurons which output values called activations. This thesis investigates if these activations can be used as a different representation of the input data and used as the input to streaming machine learning models to assist in detecting changing data and in DNN adaptation.

To address this, initially, this thesis proposes a method that handles outlier detection, adding an extra classification of ’unknown’ to DNNs, so known classes are classified as their class and the detected changed data is classified as ’unknown’. Activations are extracted, providing a unique dynamic trajectory of activations for use with a streaming machine learning clustering model to detect outliers and label them as unknown. It is shown that the DNN activations can be used as a different representation of the input data and used with streaming machine learning techniques in order to detect outliers. Experiments show that our method outperforms the other leading open-set classification methods by a minimum of 2% and a maximum of 30% on the F1-Score and is between 5 and 100 times faster.

The outlier detection method leads onto offering the second contribution, which proposes a solution that handles the scenario of novel classes appearing in a stream of data (known as concept evolution) and DNN adaptation. A novel method of extracting DNN activations is used with an accuracy volatility concept evolution detection method and DNN adaptation process. Experiments show that our method outperforms other leading methods with regards to accuracy when placed in the concept evolution scenario with limited true-labelled data. The results of the experiments are analysed based on accuracy, speed of inference and speed of adaptation. On accuracy, our method outperforms the next best adaptation method by 27% and the next best combined novel class detection and CNN adaptation method by 24%. On speed, our method is within 1.5ms of the fastest inference speed and within 1.6s of the fastest DNN adaptation speed.

Thirdly, this thesis proposes a method that handles the more advanced problem of concept drift detection and DNN adaptation in drift pattern scenarios. DNN activations from multiple hidden layers are used with our novel ensemble drift detection and DNN adaptation method. Experiments show that our method overall outperforms other leading methods of detecting concept drift and DNN adaptation in all drift scenarios. We compare with eleven other leading methods of drift detection, adaptation and combined detection and adaptation methods. Our method outperforms other leading drift detection methods by between 8% and 46% on F1-Score, and other leading drift detection and adaptation methods by between 5% and 20% on accuracy. Our method is within 1.1ms of the fastest inference speed and 7 times faster than other adaptation methods.

These three methods culminate to provide the overall contributions of this thesis, which are: (1) The application of methods to extract activations from DNNs, providing more features than the input data, termed by us as activation classification footprints; and (2) applying these footprints to our own drift detection and DNN adaptation methods in order to detect and adapt to outliers, concept evolution and concept drift. The use of DNN activations in streaming machine learning models to detect data changes and in DNN adaptation offers a unique perspective in this research area and is a step forward towards realising fully adaptive continuous deep learning systems.

Item Type:	Thesis (Doctoral)
Dates:	Date Event 21 December 2023 Submitted 12 June 2024 Accepted
Uncontrolled Keywords:	Deep neural networks, Concept evolution, Novel class detection, Concept drift, Streaming machine learning, Deep neural network adaptation
Subjects:	CAH11 - computing > CAH11-01 - computing > CAH11-01-01 - computer science CAH11 - computing > CAH11-01 - computing > CAH11-01-03 - information systems
Divisions:	Doctoral Research College > Doctoral Theses Collection Faculty of Computing, Engineering and the Built Environment > College of Computing
Depositing User:	Jaycie Carter
Date Deposited:	16 Sep 2024 10:18
Last Modified:	16 Sep 2024 10:18
URI:	https://www.open-access.bcu.ac.uk/id/eprint/15838

Actions (login required)

View Item