-
Encoder-Quantization-Motion-based Video Quality Metrics
Authors:
Yixu Chen,
Zaixi Shang,
Hai Wei,
Yongjun Wu,
Sriram Sethuraman
Abstract:
In an adaptive bitrate streaming application, the efficiency of video compression and the encoded video quality depend on both the video codec and the quality metric used to perform encoding optimization. The development of such a quality metric need large scale subjective datasets. In this work we merge several datasets into one to support the creation of a metric tailored for video compression a…
▽ More
In an adaptive bitrate streaming application, the efficiency of video compression and the encoded video quality depend on both the video codec and the quality metric used to perform encoding optimization. The development of such a quality metric need large scale subjective datasets. In this work we merge several datasets into one to support the creation of a metric tailored for video compression and scaling. We proposed a set of HEVC lightweight features to boost performance of the metrics. Our metrics can be computed from tightly coupled encoding process with 4% compute overhead or from the decoding process in real-time. The proposed method can achieve better correlation than VMAF and P.1204.3. It can extrapolate to different dynamic ranges, and is suitable for real-time video quality metrics delivery in the bitstream. The performance is verified by in-distribution and cross-dataset tests. This work paves the way for adaptive client-side heuristics, real-time segment optimization, dynamic bitrate capping, and quality-dependent post-processing neural network switching, etc.
△ Less
Submitted 9 April, 2024;
originally announced April 2024.
-
HDR or SDR? A Subjective and Objective Study of Scaled and Compressed Videos
Authors:
Joshua P. Ebenezer,
Zaixi Shang,
Yixu Chen,
Yongjun Wu,
Hai Wei,
Sriram Sethuraman,
Alan C. Bovik
Abstract:
We conducted a large-scale study of human perceptual quality judgments of High Dynamic Range (HDR) and Standard Dynamic Range (SDR) videos subjected to scaling and compression levels and viewed on three different display devices. HDR videos are able to present wider color gamuts, better contrasts, and brighter whites and darker blacks than SDR videos. While conventional expectations are that HDR q…
▽ More
We conducted a large-scale study of human perceptual quality judgments of High Dynamic Range (HDR) and Standard Dynamic Range (SDR) videos subjected to scaling and compression levels and viewed on three different display devices. HDR videos are able to present wider color gamuts, better contrasts, and brighter whites and darker blacks than SDR videos. While conventional expectations are that HDR quality is better than SDR quality, we have found subject preference of HDR versus SDR depends heavily on the display device, as well as on resolution scaling and bitrate. To study this question, we collected more than 23,000 quality ratings from 67 volunteers who watched 356 videos on OLED, QLED, and LCD televisions. Since it is of interest to be able to measure the quality of videos under these scenarios, e.g. to inform decisions regarding scaling, compression, and SDR vs HDR, we tested several well-known full-reference and no-reference video quality models on the new database. Towards advancing progress on this problem, we also developed a novel no-reference model called HDRPatchMAX, that uses both classical and bit-depth sensitive distortion statistics more accurately than existing metrics.
△ Less
Submitted 25 April, 2023;
originally announced April 2023.
-
HDR-ChipQA: No-Reference Quality Assessment on High Dynamic Range Videos
Authors:
Joshua P. Ebenezer,
Zaixi Shang,
Yongjun Wu,
Hai Wei,
Sriram Sethuraman,
Alan C. Bovik
Abstract:
We present a no-reference video quality model and algorithm that delivers standout performance for High Dynamic Range (HDR) videos, which we call HDR-ChipQA. HDR videos represent wider ranges of luminances, details, and colors than Standard Dynamic Range (SDR) videos. The growing adoption of HDR in massively scaled video networks has driven the need for video quality assessment (VQA) algorithms th…
▽ More
We present a no-reference video quality model and algorithm that delivers standout performance for High Dynamic Range (HDR) videos, which we call HDR-ChipQA. HDR videos represent wider ranges of luminances, details, and colors than Standard Dynamic Range (SDR) videos. The growing adoption of HDR in massively scaled video networks has driven the need for video quality assessment (VQA) algorithms that better account for distortions on HDR content. In particular, standard VQA models may fail to capture conspicuous distortions at the extreme ends of the dynamic range, because the features that drive them may be dominated by distortions {that pervade the mid-ranges of the signal}. We introduce a new approach whereby a local expansive nonlinearity emphasizes distortions occurring at the higher and lower ends of the {local} luma range, allowing for the definition of additional quality-aware features that are computed along a separate path. These features are not HDR-specific, and also improve VQA on SDR video contents, albeit to a reduced degree. We show that this preprocessing step significantly boosts the power of distortion-sensitive natural video statistics (NVS) features when used to predict the quality of HDR content. In similar manner, we separately compute novel wide-gamut color features using the same nonlinear processing steps. We have found that our model significantly outperforms SDR VQA algorithms on the only publicly available, comprehensive HDR database, while also attaining state-of-the-art performance on SDR content.
△ Less
Submitted 25 April, 2023;
originally announced April 2023.
-
Making Video Quality Assessment Models Robust to Bit Depth
Authors:
Joshua P. Ebenezer,
Zaixi Shang,
Yongjun Wu,
Hai Wei,
Sriram Sethuraman,
Alan C. Bovik
Abstract:
We introduce a novel feature set, which we call HDRMAX features, that when included into Video Quality Assessment (VQA) algorithms designed for Standard Dynamic Range (SDR) videos, sensitizes them to distortions of High Dynamic Range (HDR) videos that are inadequately accounted for by these algorithms. While these features are not specific to HDR, and also augment the equality prediction performan…
▽ More
We introduce a novel feature set, which we call HDRMAX features, that when included into Video Quality Assessment (VQA) algorithms designed for Standard Dynamic Range (SDR) videos, sensitizes them to distortions of High Dynamic Range (HDR) videos that are inadequately accounted for by these algorithms. While these features are not specific to HDR, and also augment the equality prediction performances of VQA models on SDR content, they are especially effective on HDR. HDRMAX features modify powerful priors drawn from Natural Video Statistics (NVS) models by enhancing their measurability where they visually impact the brightest and darkest local portions of videos, thereby capturing distortions that are often poorly accounted for by existing VQA models. As a demonstration of the efficacy of our approach, we show that, while current state-of-the-art VQA models perform poorly on 10-bit HDR databases, their performances are greatly improved by the inclusion of HDRMAX features when tested on HDR and 10-bit distorted videos.
△ Less
Submitted 25 April, 2023;
originally announced April 2023.
-
Subjective Assessment of High Dynamic Range Videos Under Different Ambient Conditions
Authors:
Zaixi Shang,
Joshua P. Ebenezer,
Alan C. Bovik,
Yongjun Wu,
Hai Wei,
Sriram Sethuraman
Abstract:
High Dynamic Range (HDR) videos can represent a much greater range of brightness and color than Standard Dynamic Range (SDR) videos and are rapidly becoming an industry standard. HDR videos have more challenging capture, transmission, and display requirements than legacy SDR videos. With their greater bit depth, advanced electro-optical transfer functions, and wider color gamuts, comes the need fo…
▽ More
High Dynamic Range (HDR) videos can represent a much greater range of brightness and color than Standard Dynamic Range (SDR) videos and are rapidly becoming an industry standard. HDR videos have more challenging capture, transmission, and display requirements than legacy SDR videos. With their greater bit depth, advanced electro-optical transfer functions, and wider color gamuts, comes the need for video quality algorithms that are specifically designed to predict the quality of HDR videos. Towards this end, we present the first publicly released large-scale subjective study of HDR videos. We study the effect of distortions such as compression and aliasing on the quality of HDR videos. We also study the effect of ambient illumination on perceptual quality of HDR videos by conducting the study in both a dark lab environment and a brighter living-room environment. A total of 66 subjects participated in the study and more than 20,000 opinion scores were collected, which makes this the largest in-lab study of HDR video quality ever. We anticipate that the dataset will be a valuable resource for researchers to develop better models of perceptual quality for HDR videos.
△ Less
Submitted 20 September, 2022;
originally announced September 2022.
-
Subjective and Objective Quality Assessment of High-Motion Sports Videos at Low-Bitrates
Authors:
Joshua P. Ebenezer,
Yixu Chen,
Yongjun Wu,
Hai Wei,
Sriram Sethuraman
Abstract:
Videos often have to be transmitted and stored at low bitrates due to poor network connectivity during adaptive bitrate streaming. Designing optimal bitrate ladders that would select the perceptually-optimized resolution, frame-rate, and compression level for low-bitrate videos for adaptive streaming across the internet is therefore a task of great interest. Towards that end, we conducted the firs…
▽ More
Videos often have to be transmitted and stored at low bitrates due to poor network connectivity during adaptive bitrate streaming. Designing optimal bitrate ladders that would select the perceptually-optimized resolution, frame-rate, and compression level for low-bitrate videos for adaptive streaming across the internet is therefore a task of great interest. Towards that end, we conducted the first large-scale study of medium and low-bitrate videos from live sports for two codecs (Elemental AVC and HEVC) and created the Amazon Prime Video Low-Bitrate Sports (APV LBS) dataset. The study involved 94 participants and 742 videos, with more than 23,000 human opinion scores collected in total. We analyzed the data obtained and we also conducted an extensive evaluation of objective Video Quality Assessment (VQA) algorithms and benchmarked their performance, and make recommendations on bitrate ladder design. We're making the metadata and VQA features available at https://github.com/JoshuaEbenezer/lbmfr-public.
△ Less
Submitted 12 July, 2022;
originally announced July 2022.
-
On the benefit of parameter-driven approaches for the modeling and the prediction of Satisfied User Ratio for compressed video
Authors:
Jingwen Zhu,
Patrick Le Callet,
Anne-Flore Perrin,
Sriram Sethuraman,
Kumar Rahul
Abstract:
The human eye cannot perceive small pixel changes in images or videos until a certain threshold of distortion. In the context of video compression, Just Noticeable Difference (JND) is the smallest distortion level from which the human eye can perceive the difference between reference video and the distorted/compressed one. Satisfied-User-Ratio (SUR) curve is the complementary cumulative distributi…
▽ More
The human eye cannot perceive small pixel changes in images or videos until a certain threshold of distortion. In the context of video compression, Just Noticeable Difference (JND) is the smallest distortion level from which the human eye can perceive the difference between reference video and the distorted/compressed one. Satisfied-User-Ratio (SUR) curve is the complementary cumulative distribution function of the individual JNDs of a viewer group. However, most of the previous works predict each point in SUR curve by using features both from source video and from compressed videos with assumption that the group-based JND annotations follow Gaussian distribution, which is neither practical nor accurate. In this work, we firstly compared various common functions for SUR curve modeling. Afterwards, we proposed a novel parameter-driven method to predict the video-wise SUR from video features. Besides, we compared the prediction results of source-only features based (SRC-based) models and source plus compressed videos features (SRC+PVS-based) models.
△ Less
Submitted 20 June, 2022;
originally announced June 2022.
-
ChipQA: No-Reference Video Quality Prediction via Space-Time Chips
Authors:
Joshua P. Ebenezer,
Zaixi Shang,
Yongjun Wu,
Hai Wei,
Sriram Sethuraman,
Alan C. Bovik
Abstract:
We propose a new model for no-reference video quality assessment (VQA). Our approach uses a new idea of highly-localized space-time (ST) slices called Space-Time Chips (ST Chips). ST Chips are localized cuts of video data along directions that \textit{implicitly} capture motion. We use perceptually-motivated bandpass and normalization models to first process the video data, and then select oriente…
▽ More
We propose a new model for no-reference video quality assessment (VQA). Our approach uses a new idea of highly-localized space-time (ST) slices called Space-Time Chips (ST Chips). ST Chips are localized cuts of video data along directions that \textit{implicitly} capture motion. We use perceptually-motivated bandpass and normalization models to first process the video data, and then select oriented ST Chips based on how closely they fit parametric models of natural video statistics. We show that the parameters that describe these statistics can be used to reliably predict the quality of videos, without the need for a reference video. The proposed method implicitly models ST video naturalness, and deviations from naturalness. We train and test our model on several large VQA databases, and show that our model achieves state-of-the-art performance at reduced cost, without requiring motion computation.
△ Less
Submitted 17 September, 2021;
originally announced September 2021.
-
Assessment of Subjective and Objective Quality of Live Streaming Sports Videos
Authors:
Zaixi Shang,
Joshua P. Ebenezer,
Alan C. Bovik,
Yongjun Wu,
Hai Wei,
Sriram Sethuraman
Abstract:
Video live streaming is gaining prevalence among video streaming services, especially for the delivery of popular sporting events. Many objective Video Quality Assessment (VQA) models have been developed to predict the perceptual quality of videos. Appropriate databases that exemplify the distortions encountered in live streaming videos are important to designing and learning objective VQA models.…
▽ More
Video live streaming is gaining prevalence among video streaming services, especially for the delivery of popular sporting events. Many objective Video Quality Assessment (VQA) models have been developed to predict the perceptual quality of videos. Appropriate databases that exemplify the distortions encountered in live streaming videos are important to designing and learning objective VQA models. Towards making progress in this direction, we built a video quality database specifically designed for live streaming VQA research. The new video database is called the Laboratory for Image and Video Engineering (LIVE) Live stream Database. The LIVE Livestream Database includes 315 videos of 45 contents impaired by 6 types of distortions. We also performed a subjective quality study using the new database, whereby more than 12,000 human opinions were gathered from 40 subjects. We demonstrate the usefulness of the new resource by performing a holistic evaluation of the performance of current state-of-the-art (SOTA) VQA models. The LIVE Livestream database is being made publicly available for these purposes at https://live.ece.utexas.edu/research/LIVE_APV_Study/apv_index.html.
△ Less
Submitted 15 June, 2021;
originally announced June 2021.
-
Detection of Audio-Video Synchronization Errors Via Event Detection
Authors:
Joshua P. Ebenezer,
Yongjun Wu,
Hai Wei,
Sriram Sethuraman,
Zongyi Liu
Abstract:
We present a new method and a large-scale database to detect audio-video synchronization(A/V sync) errors in tennis videos. A deep network is trained to detect the visual signature of the tennis ball being hit by the racquet in the video stream. Another deep network is trained to detect the auditory signature of the same event in the audio stream. During evaluation, the audio stream is searched by…
▽ More
We present a new method and a large-scale database to detect audio-video synchronization(A/V sync) errors in tennis videos. A deep network is trained to detect the visual signature of the tennis ball being hit by the racquet in the video stream. Another deep network is trained to detect the auditory signature of the same event in the audio stream. During evaluation, the audio stream is searched by the audio network for the audio event of the ball being hit. If the event is found in audio, the neighboring interval in video is searched for the corresponding visual signature. If the event is not found in the video stream but is found in the audio stream, A/V sync error is flagged. We developed a large-scaled database of 504,300 frames from 6 hours of videos of tennis events, simulated A/V sync errors, and found our method achieves high accuracy on the task.
△ Less
Submitted 20 April, 2021;
originally announced April 2021.
-
MyWear: A Smart Wear for Continuous Body Vital Monitoring and Emergency Alert
Authors:
Sibi C. Sethuraman,
Pranav Kompally,
Saraju P. Mohanty,
Uma Choppali
Abstract:
Smart healthcare which is built as healthcare Cyber-Physical System (H-CPS) from Internet-of-Medical-Things (IoMT) is becoming more important than before. Medical devices and their connectivity through Internet with alongwith the electronics health record (EHR) and AI analytics making H-CPS possible. IoMT-end devices like wearables and implantables are key for H-CPS based smart healthcare. Smart g…
▽ More
Smart healthcare which is built as healthcare Cyber-Physical System (H-CPS) from Internet-of-Medical-Things (IoMT) is becoming more important than before. Medical devices and their connectivity through Internet with alongwith the electronics health record (EHR) and AI analytics making H-CPS possible. IoMT-end devices like wearables and implantables are key for H-CPS based smart healthcare. Smart garment is a specific wearable which can be used for smart healthcare. There are various smart garments that help users to monitor their body vitals in real-time. Many commercially available garments collect the vital data and transmit it to the mobile application for visualization. However, these don't perform real-time analysis for the user to comprehend their health conditions. Also, such garments are not included with an alert system to alert users and contacts in case of emergency. In MyWear, we propose a wearable body vital monitoring garment that captures physiological data and automatically analyses such heart rate, stress level, muscle activity to detect abnormalities. A copy of the physiological data is transmitted to the cloud for detecting any abnormalities in heart beats and predict any potential heart failure in future. We also propose a deep neural network (DNN) model that automatically classifies abnormal heart beat and potential heart failure. For immediate assistance in such a situation, we propose an alert system that sends an alert message to nearby medical officials. The proposed MyWear has an average accuracy of 96.9% and precision of 97.3% for detection of the abnormalities.
△ Less
Submitted 17 October, 2020;
originally announced October 2020.
