JPEG XR objective and subjective evaluations

Rate-distortion p erformance evaluation of JPEG-XR Objective results and proposed methodology for subjective quality assessment Francesca De Simone, Frederic Dufaux, Touradj Ebrahimi JPEG XR Camera Raw coding AHG meeting 18-19 February 2009, Tokyo, Japan

Introduction Quality Assessment (QA) and codec performance evaluation Status Our previous contributions Objective QA Test material Codecs and configuration parameters Quality metrics Selected results Subjective QA Proposed methodology Test conditions Preliminary results Outline of the presentation

Introduction Quality Assessment (QA) and codec performance evaluation Status Our previous contributions

Codec performance evaluation in terms of: Compression efficiency. Computational requirements. Additional functionalities. QA and codec performance evaluation Rate-Distortion (RD) curves = quality measure vs bit per pixel Original picture Output picture JPEG or JPEG 2000 or JPEG XR HUMAN SUBJECT (subjective QA) or FR METRIC (objective QA)

THERE ARE NOT YET RELIABLE and STANDARD OBJECTIVE METHODS FOR IMAGE QUALITY ASSESSMENT Image and video systems complexity Human Visual System (HVS) complexity Lack of standardization Objective QA can be performed to provide a first comparison of a wide range of conditions. Subjective QA needs to be performed as benchmark, to validate the results of the objective metrics. Status

JPEG contributions: F. De Simone et al., “Comparison of PSNR performance of HD Photo and JPEG2000” , wg1n4404, JPEG meeting Kobe (Nov. 2007) F. De Simone et al., “Objective evaluation of the rate‐distortion performance of JPEG‐XR” , wg1n4552, JPEG Interim meeting Poitiers (Feb. 2008) F. De Simone et al., “ Still image coding algorithms performance comparison: objective quality metrics ” , wg1n4497, JPEG meeting San Francisco (Apr. 2008) F. De Simone et al., “Objective rate‐distortion performance of different JPEG‐XR implementations” , wg1n4701, JPEG meeting Poitiers (July 2008) Conference publications: F. De Simone et al., “ A comparative study of JPEG 2000, AVC/H.264, and HD Photo ” , SPIE Optics and Photonics, Applications of Digital Image Processing XXX, 6696 (Aug. 2007) F. De Simone et al., “A comparative study of color image compression standards using perceptually driven quality metrics” , SPIE Optics and Photonics, Applications of Digital Image Processing XXXI (Aug. 2008) Our previous contributions

Objective QA Test material Codecs and configuration parameters Quality metrics Selected results

Test Material – 24 bpp pictures (sample pictures from Thomas Richter dataset, 2 different spatial resolutions: 3888x2592, 2592x3888 ) (sample pictures from Microsoft dataset, 6 different spatial resolutions: 4064x2704, 2268x1512, 2592x1944, 2128x2832, 2704x3499, 4288x2848)

JPEG XR vs JPEG2000 vs JPEG: JPEG XR (DPK version 1.0): one level overlapping and two level overlapping. 4:4:4 and 4:2:0 chroma subsampling. JPEG 2000 (Kakadu version 6.0): default settings (64x64 code-block size, 1 quality layer, no precincts, 1 tile, 9x7 wavelet, 5 decomposition levels). rate control. no visual frequency weighting and visual frequency weighting. 4:4:4 and 4:2:0 chroma subsampling. JPEG (IJG version 6b ): default settings (Huffman coding). default visually optimized quantization tables. 4:4:4 and 4:2:0 chroma subsampling. Codecs and configuration parameters

Different JPEG XR implementations: JPEG XR DPK version 1.0: different quantization steps for different color channels (default). same quantization steps for different frequency bands (default). JPEG XR Reference Software version 1.0: same quantization steps for different color channels (default). same quantization steps for different frequency bands (default). JPEG XR Reference Software version 1.2 ‐ i.e. Thomas Ricther’s version: different quantization steps for different color channels (same as DPK). different quantization steps for different frequency bands (default). new POT (leakage fix described in wg1n4660) (default). JPEG XR Microsoft implementation described in HDPn21 / wg1n4549 : different quantization steps for different color channels (enhanced encoding techniques described in HDPn21 / wg1n4549) (default). different quantization steps for different frequency bands (enhanced encoding techniques of HDPn21 / wg1n4549) (default). new POT (leakage fix described in wg1n4660) (default). Codecs and configuration parameters

Metric 1: Maximum Pixel Deviation (L inf ) L inf R = max [abs(Im aR (x,y)-Im bR (x,y))] Considering RGB color space: where: Im a , Im b = pictures to compare L inf G = max [abs(Im aG (x,y)-Im bG (x,y))] L inf B = max [abs(Im aB (x,y)-Im bB (x,y))] (L inf  [0,1])

PSNR evaluation considering: R, G and B components Y’, C b and C r components (ITU-R Rec. BT.601) where: M, N = image dimensions Im a , Im b = pictures to compare B= bit depth Metric 2: single channel PSNR

Metric 3: PSNR weighted average (WPSNR) WPSNR = w 1 PSNR 1 + w 2 PSNR 2 + w 3 PSNR 3 PSNR considering weighted summation of the PSNRs evaluated on R, G and B components or Y’, Cb and Cr components (ITU-R Rec. BT.601): where: , considering R,G, and B components. , considering Y’, C b , and C r components.

Metric 3: PSNR weighted average (WPSNR_MSE) WPSNR_MSE PSNR considering weighted summation of the MSEs evaluated on R, G and B components or Y’, Cb and Cr components (ITU-R Rec. BT.601): where: , considering R,G, and B components. , considering Y’, C b , and C r components.

Metric 3: PSNR weighted average (WPSNR_PIX) WPSNR_PIX PSNR considering MSE evaluated on weighted summation of the image R, G and B components : where: M, N = image dimensions Im a , Im b = pictures to compare B= bit depth , considering R,G, and B components. , considering Y’, Cb, and Cr components.

Estimate of luminance = mean intensity: Metric 4: Mean SSIM (MSSIM) (I) Estimate of contrast = standard deviation: [1] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “ Image Quality Assessment: From Error Measurement to Structural Similarity” (2004). Estimate of picture structure: Structural information = “attributes that represent the structure of objects in the scene, independent of the average luminance and contrast”.

Metric 4: Mean SSIM (MSSIM) (II) Luminance comparison function: (C 1 =constant) Contrast comparison function: (C 2 =constant) Measure of structural similarity = correlation between and Structure comparison function: where (C 3 =constant)

Metric 4: Mean SSIM (MSSIM) (III) The SSIM indexing algorithm is applied using a sliding window approach which results in a SSIM index quality map of the image. The average of the quality map is called Mean SSIM index (MSSIM) . Weighted summation of MSSIM indexes evaluated on Y’, Cb and Cr components ( Y’CbCr color space - Rec. ITU-R BT.601 ): MSSIM = w y MSSIM Y + w Cb MSSIM Cb + w Cr MSSIM Cr where: (MSSIM  [0,1])

Metric 5: Visual Information Fidelity – Pixel (VIF-P) (I) [2] H. R. Sheikh, A. C. Bovik “Image Information And Visual Quality” (2004). “ Image information measure that quantifies the information that is present in the reference image and how much this reference information can be extracted from the distorted image” using statistical approach. Natural image (source) Channel (distortion) HVS HVS C F E Reference image (E) = output of a stochastic natural source that passes through HVS channel and is processed by the brain Test image (F) = output of an image distortion channel that distorts the output of the natural source before it passes through the HVS channel

Metric 5: Visual Information Fidelity – Pixel (VIF-P) (II) Natural image modeling in wavelet domain using Gaussian scale mixtures (GSMs) Information that the brain could ideally extract from reference image = mutual information between C and E: Corresponding information that could be extracted from test image = mutual information between C and F: VIF-P is a new implementation in a multi-scale pixel domain: computationally simpler than Wavelet domain version. performance slightly worse than Wavelet domain version. where: z= source model parameters. (VIF  [0,1] and VIF>1 if the test image is enhanced version of the original )

Metric 6: PSNR-HVS-M (I) Block 8x8 of distorted image Block 8x8 of original image DCT of difference between pixel values Reduction by value of contrast masking MSE H calculation of the block [3] N. Ponomarenko, F. Silvestri, K. Egiazarian, M.Carli, J. Astola, and V. Lukin, “ On between-coefficient contrast masking of DCT basis functions” (2007). DCT coefficients of 8x8 pixel blocks X and Y are visually undistinguished if: E w (X-Y) < max (E m (X), E m (Y)) where E w (block) is the energy of DCT coefficients of the block weighted according to CSF and E m (block) is the masking effect of DCT coefficients of the block which depends upon E w (block) and upon the local variances.

Metric 6: PSNR-HVS-M (II) where: M, N = image dimensions K= constant = visible difference between DCT coefficient of the original image and distorted image 8x8 blocks , depending upon contrast masking T c = matrix of correcting factors based on standard visually optimized JPEG quantization tables B= bit depth

Metric 7: DC Tune [4] A. B. Watson, A. P. Gale, J. A. Solomon, and A. J. Ahumada JR., “ DCTune: A Techinque For Visual Optimization Of DCT Quantization Matrices For Individual Images” (1994). developed as a method for optimizing JPEG image compression by computing the JPEG quantization matrices which yields a designated perceptual error model of perceptual error based upon DCT coefficients analysis, taking into account: luminance masking. contrast masking. spatial error pooling. frequency error pooling.

Selected results 4:4:4 – JPEG XR vs JPEG2000 vs JPEG Average over image dataset of PSNR values on R component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) on G component: on B component:

Average over image dataset of PSNR values on Y’ component: on Cb component: on Cr component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) Selected results 4:4:4 – JPEG XR vs JPEG2000 vs JPEG

Average over image dataset of WPSNR values on Y’CbCr components: on RGB components: bpp (bits/pixel) bpp (bits/pixel) Selected results 4:4:4 – JPEG XR vs JPEG2000 vs JPEG

Average over image dataset of WPSNR-MSE values on Y’CbCr components: on RGB components: bpp (bits/pixel) bpp (bits/pixel) Selected results 4:4:4 – JPEG XR vs JPEG2000 vs JPEG

Average over image dataset of WPSNR-PIX values on Y’CbCr components: on RGB components: bpp (bits/pixel) bpp (bits/pixel) Selected results 4:4:4 – JPEG XR vs JPEG2000 vs JPEG

Average over image dataset of MSSIM values on Y’ component: on Cb component: on Cr component: bpp (bits/pixel) Selected results 4:4:4 – JPEG XR vs JPEG2000 vs JPEG

Average over image dataset of VIF-P values on Y’ component only: bpp (bits/pixel) Selected results 4:4:4 – JPEG XR vs JPEG2000 vs JPEG

Average over image dataset of PSNR values (one level POT) on R component: on G component: on B component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) Selected results 4:4:4 – different JPEG XR implem.

Average over image dataset of PSNR values (two levels POT) on R component: on G component: on B component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) Selected results 4:4:4 – different JPEG XR implem.

Average over image dataset of PSNR values (one level POT) on Y component: on Cb component: on Cr component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) Selected results 4:4:4 – different JPEG XR implem.

Average over image dataset of PSNR values (two levels POT) on Y component: on Cb component: on Cr component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) Selected results 4:4:4 – different JPEG XR implem.

Average over image dataset of WPSNR_MSE values bpp (bits/pixel) bpp (bits/pixel) one level POT: two levels POT: Selected results 4:4:4 – different JPEG XR implem.

Average over image dataset of MSSIM values (one level POT) on Y component: on Cb component: on Cr component: bpp (bits/pixel) Selected results 4:4:4 – different JPEG XR implem.

Average over image dataset of MSSIM values (two levels POT) on Y component: on Cb component: on Cr component: bpp (bits/pixel) Selected results 4:4:4 – different JPEG XR implem.

Subjective QA Proposed methodology Test conditions Preliminary results

Double Stimulus Continuous Quality Scale (DSCQS) method [ITU-R Rec. BT.500-11] adapted to deal with the evaluation of still pictures: Proposed methodology (I) Reference Image Test Image test picture and its reference are shown at the same time. the assessor is not told about the presence of a reference picture. positions of reference and test pictures are systematically switched. test pairs related to different original contents are always alternated.

Proposed methodology (II) when the subject clicks into the active area of the screen a rating window is shown:

Proposed methodology (III) the subject has to rate the quality of the two pictures choosing for each a value in between 0 (worse quality possible) to 100 (best quality possible). Rating window (Continuous Quality Scale )

Proposed methodology (IV) Subjects are checked for visual acuity and color blindness Before each session, instructions are provided to subjects and a training session is performed to explain how to use the rating scale contents shown for training are not used for testing data gathered during the training are not included in the final test results Some dummy presentations are inserted at the beginning of the test to stabilize subject’s behaviour data gathered from the dummies are not included in the final test results the dummy presentations cover all the quality levels included in the test material The test session lasts no more than 20 minutes (including training)

Proposed methodology (V) At least 15 subjects Subjective data processing: computation of Differential Score (DS) : DS = Score for the reference picture – Score for the test picture ANalysis Of Variance (ANOVA) to detect eventual systematic errors and scores normalization to remove them screening to detect outliers [ITU-R Rec. BT.500-11] computation of the Differential Mean Opinion Score (DMOS)

Test conditions Eizo CG301W LCD monitor (2560x1600 pixels) monitor calibration using color calibration device (EyeOne Display2) Gamut sRGB, white point D65, brightness 120cd/m2, minimum black level. controlled lighting system: neon lamps with 6500 K color temperature ambient light measurement by EyeOne Display2 tool

Preliminary results (I) JPEG XR Microsoft implementation described in HDPn21: different quantization steps for different color channels (enhanced encoding techniques described in HDPn21 / wg1n4549) (default) different quantization steps for different frequency bands (enhanced encoding techniques of HDPn21 / wg1n4549) (default) new POT (leakage fix described in wg1n4660) (default) 4:4:4 coding, one level POT 4 contents, 7 selected samples corresponding to the following bpp values: Content q=40 (T1) q=50 (T2) q=58 (T3) q=66 (T4) q=76 (T5) q=82 (T6) q=90 (T7) Cont. 1 0.9 0.64 0.46 0.34 0.22 0.18 0.13 Cont. 2 0.15 0.1 0.07 0.05 0.04 0.03 0.02 Cont. 3 0.9 0.61 0.43 0.31 0.19 0.15 0.1 Cont. 4 0.65 0.44 0.31 0.22 0.13 0.09 0.06

Preliminary results (II) 2 contents, other than those used in the test session, have been used for the training session 17 subjects have taken part to the experiment: 3 females, 14 males average subject’s age 29 Statistical analysis of the data: inter-subjects ANOVA offset and gain score normalization outliers’ screening: 4 outliers for content 1 2 outliers for content 2 2 outliers for content 3 5 outliers for content 4

Acknowledgement Part of the work reported here has been possible thanks to: European Commission funded Network of Excellence on ‘Networked Audiovisual Media Technologies VISNET II’

Thank you for your attention! Questions?

JPEG XR objective and subjective evaluations

Recommended

More Related Content

What's hot (20)

Viewers also liked (14)

Similar to JPEG XR objective and subjective evaluations (20)

More from Touradj Ebrahimi (20)

Recently uploaded (20)

JPEG XR objective and subjective evaluations