Big data flow processing.
Big data flow processing.
Many hardware-based techniques have been developed for support of increasing data flows: high-speed network channels and memory buses, high frequency CPUs, hard disks with high data density and low access time. However, numerous unsolved problems remain on the software side dealing with processing, analyzing and storing data. This software must use hardware resources efficiently and also satisfy rigid requirements: support batch processing of huge data volumes with high throughput, provide reliable functioning on unreliable hardware, allow for good scaling and efficient random data access. This project is aimed at creating a framework for data acquirement, filtering, analysis and storage in real time on high-speed network channels. This framework will allow automation of a wide range of tasks related to high-speed data flows: classifying traffic, ensuring network security, analyzing social networks, and forecasting using big data.
