Video Analytics Trends Part 5: Intelligent Storage Management

Graeme Woods

Global Business Analyst

This is the fifth part of this series on trends in video analytics. It is focused on the storage of video and associated data.

Previous parts have discussed how advances in AI, cloud and edge processing are transforming surveillance by increasing business value outside of traditional security cases and extending it to areas such as retail analytics and smart cities.

In retail analytics, we analyse customer movement through a retail space, where they spend time in the store, and dwell and queuing times.

In smart cities, surveillance extends beyond security to help manage traffic, reduce energy consumption and pollution, and optimise infrastructure. The proposed Neom development in Saudi Arabia is a vision of how AI can be potentially used to manage and optimize an entire city.

This advanced analysis requires access to large quantities of both real-time, historical video data and associated data, including labels and detections applied to video data. This is because a lot of the analysis required to identify patterns and causal relationships is based on a time series.

There are other technology trends that impact on data storage:

● Deployments are using a larger number of cameras with higher resolution. Where 2K cameras were common, now 8K cameras are being deployed. This means more data needs to be transmitted, processed, and stored.

● Improved video compression is increasingly used. H.265 offers comparable video quality to H.264 with half the file size. This makes both data transmission and storage more practical and cost effective.

● Cloud storage and bandwidth costs are reducing, while cloud uptake is increasing. Combined with simpler provisioning with no upfront capital costs, this is encouraging video surveillance users to move from on-premises to cloud installations.

● Edge devices can filter data prior to transmission, saving bandwidth and storage. Whilst some users want to store all data, edge devices can preprocess data and significantly reduce the amount that must be uploaded or stored.

The following diagram shows the interactions between these trends for storage of video and detections:

Video surveillance users either store video selectively (such as when there is movement or a detection), or just store everything, relying on compression to reduce storage and bandwidth costs. As storage and bandwidth costs decrease, it is becoming more feasible to store all video and make it available for offline analysis.

IPVM conducted a survey in 2021 that indicates that most users (60 percent) store data for 30 days. Only 9% of users store data for three months or more. Longer periods are generally associated with government or government regulation. One approach is to store all data for 30 days, then selectively store data for longer periods to support regulatory requirements or time series analysis, subject to privacy regulations.

As I discussed in part four of this series, traditionally, video data has been stored in on-premises servers. As well as the cost of buying, installing and hosting servers with the required storage, on-premises sites also need to factor in the costs of offsite backup of critical data.

Cloud processing is rapidly replacing on-premises installations due to elimination of capital costs, simpler provisioning and management, and the same is true for cloud storage. Cloud services such as AWS S3 (an object store) offer virtually unlimited quantities of highly durable and available storage. S3 has a durability design target of 99.99999999999%, which is equivalent to losing one out of 10 million stored objects, once every 10,000 years.

For long term data archiving, video can be selectively moved from fast online storage to slower, cheaper cold storage such as AWS Glacier. Google Cloud and Azure offer comparable solutions for object storage.

It is very easy to back up data across multiple cloud availability zones.

To summarise, video surveillance is moving beyond security use cases and is adding increasing business value to other areas, but this requires storage of more data, especially to allow analysis of time series. Technology improvements in camera resolution, compression algorithms and cloud / edge processing are changing the amount of data available and how it is stored. There are clear trends towards filtering at the edge and then storing data in cloud object stores, and having that data available to support analysis, before selectively migrating it to cold storage.