Large scale developments in modern Internet networks are not too common compared to the number of in-house solutions. We know little about ongoing developments in telecommunication in Japan, which is weird, because revolutionary solution do appear in the country almost on a daily basis.
Not strictly related to networks, but several steps of semiconductor design is Japanese monopoly and even though many semiconductor factories are in China, these places cannot operate without obtaining Japanese consent. Besides semiconductors Japan is pioneering in the field of modern Internet networks.
The solution I am going to introduce here is being developed by one of the largest IT firm in Japan, NTT.
Deep learning is one of the most popular fields of modern science and in spite of its obscure nature the number of scientists researching and developing deep learning based solutions is undoubtedly increasing. We witness terms such as machine learning, image processing and artificial intelligence as well and these are very important technologies when it comes to developing network security solutions.
The Japan-based NTT is working on such a system that is capable of detecting data transmission failures between certain network devices.
What do we call failure in this context?
A very banal example would be when a cable gets damaged and data transmission is not possible anymore. When this happens the failure is immediate between two network devices and it can cause sever losses in the network.
Also, sometimes network devices get infected by malwares and they start emitting large amount of garbage, which can result in similar undesired losses. These should be detected and repaired as soon as possible.
Detecting failures in the network
The traditional failure detection solution are able to detect the majority of these flaws, however there are certain types of failures that are very difficult to discern using these old methods.
These traditional methods utilize some kind of threshold value and whenever a certain signals amplitude falls bellow this threshold the system alerts us. However, in some case even though there is a failure in the network, but as the amplitude is still higher than the threshold value we get no alert from the failure detection system.
So how do we set the threshold value?
For this sensitive operation, we will need several years of experience in the field of computer networks.
If we have many years of historical data available from the target system setting this threshold value is not very difficult.
Japanese technology to detect network failures
According to Japanese scientists, with the technology they are developing, these network failures can be detected in an initial stage.
The module used for failure detection has three operating modes.
In the first stage, they analyze the output data of various network devices and they create an input data object.
In this stage a neural network is utilized. The data object created in the first step is processed by a neural network. This is basically an internal layer between the input and the output data.
Or in other words, this is the learning stage.
We can call this "a learning stage", because this internal layer is being thought to be able to discern signal structures during normal operation.
So what happens when we try to input abnormal sign structures?
Such a signal differs from normal signal significantly, therefore the inner layer will be unable to process it.
Signals during normal operation are characterized by a term called "equilibrium state" and these are compared to the output signals, whence a total difference is calculated.
This measure will give the "abnormality" of a given signal.
For example, when one introduces a failure detection threshold and says that there is a failure in the network whenever the amplitude of a the signal falls under this value, in some cases he/she will not be able to detect failures because the total amplitude can exceed the threshold even though a tiny failure has appeared in the network. Treating these relative differences can be extremely difficult. In other scenarios, if the number of network users decreases for example at night, the amplitude of the signal can fall bellow the prescribed threshold and our system will give a false alarm.
The system developed by the Japanese NTT can operate in a much more sophisticated fashion.
If I wanted to phrase it simply, instead of considering amplitude changes in the network, they try to detect abnormal signal structures and for that they utilize a special neural network.
With this system, NTT is capable of detecting network failures regardless of the current number of users. Clearly, when using such a system signal amplitude is not important anymore.
Whenever the failure is detected, it would be expedient to find the source of it.
For this purpose, they utilize big data that was used previously in the learning stage.
This is possible, because they assign a weight to every single data source and this should represent the data source's contribution to the total signal.
With this parameter, it is possible to trace the failure back to the network device.
NTT is planing to use this deep learning based system in their IoT world and they are certain that they will see these systems being introduced in their factories in the very near future.