[Bug]: Previews of HDR videos being generated are washed out (needs tonemap?)

[invario]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Recently I switched on the HDR10+ setting for recording videos on my Samsung GS23+ and now previews being generated in NC looked washed out. (See still images attached of the generated preview versus a screenshot from the video being played.) From my understanding of HDR, this might be a result of needing to tonemap with some additional switches passed to FFMPEG?

NC generated preview:

Screenshot out of original video:

Steps to reproduce

1. Record video on a device using HDR
2. Upload the video to NC
3. Perform any function that causes NC to generate a preview of the HDR video

Expected behavior

The preview/thumbnail shouldn't be washed out.

Installation method

Community Manual installation with Archive

Nextcloud Server version

28

Operating system

Other

PHP engine version

PHP 8.2

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "apps_paths": [
            {
                "path": "\/usr\/local\/www\/nextcloud\/apps",
                "url": "\/apps",
                "writable": true
            }
        ],
        "defaultapp": "memories",
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "loglevel": 2,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "filelocking.enabled": true,
        "trusted_domains": [
            "localhost",
            "192.168.0.12",
            "nextcloud.MYDOMAIN.com"
        ],
        "default_phone_region": "US",
        "enable_previews": true,
        "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
        "preview_format": "jpeg",
        "enabledPreviewProviders": [
            "OC\\Preview\\Imaginary",
            "OC\\Preview\\MKV",
            "OC\\Preview\\MP3",
            "OC\\Preview\\MP4",
            "OC\\Preview\\AVI",
            "OC\\Preview\\Movie"
        ],
        "cropimagepreviews": "0",
        "preview_max_x": 2048,
        "preview_max_y": 2048,
        "jpeg_quality": "60",
        "logtimezone": "America\/New_York",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "version": "28.0.4.1",
        "updater.release.channel": "stable",
        "overwrite.cli.url": "https:\/\/nextcloud.MYDOMAIN.com",
        "dbtype": "mysql",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "maintenance": false,
        "trashbin_retention_obligation": "30,31",
        "theme": "",
        "app_install_overwrite": [
            "customproperties"
        ],
        "memories.vod.path": "\/usr\/local\/www\/nextcloud\/apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.vod.ffmpeg": "\/usr\/local\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/local\/bin\/ffprobe",
        "memories.exiftool": "\/usr\/local\/bin\/exiftool",
        "data-fingerprint": "cf5cf1b31d5af06404c30a5805d00eda",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "memories.exiftool_no_local": true,
        "memories.vod.external": true,
        "maintenance_window_start": 8,
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "memories.db.triggers.fcu": true
    }
}

List of activated Apps

Enabled:
  - activity: 2.20.0
  - admin_audit: 1.18.0
  - bruteforcesettings: 2.8.0
  - cloud_federation_api: 1.11.0
  - dav: 1.29.1
  - federatedfilesharing: 1.18.0
  - files: 2.0.0
  - files_external: 1.20.0
  - files_pdfviewer: 2.9.0
  - files_reminders: 1.1.0
  - files_sharing: 1.20.0
  - files_trashbin: 1.18.0
  - logreader: 2.13.0
  - lookup_server_connector: 1.16.0
  - memories: 7.2.0
  - notifications: 2.16.0
  - oauth2: 1.16.3
  - password_policy: 1.18.0
  - photos: 2.4.0
  - previewgenerator: 5.5.0
  - privacy: 1.12.0
  - provisioning_api: 1.18.0
  - serverinfo: 1.18.0
  - settings: 1.10.1
  - sharebymail: 1.18.0
  - survey_client: 1.16.0
  - theming: 2.3.0
  - twofactor_admin: 4.5.0
  - twofactor_backupcodes: 1.17.0
  - twofactor_totp: 10.0.0-beta.2
  - updatenotification: 1.18.0
  - viewer: 2.2.0
  - workflowengine: 2.10.0
Disabled:
  - circles: 28.0.0 (installed 22.1.1)
  - comments: 1.18.0 (installed 1.10.0)
  - contactsinteraction: 1.9.0 (installed 1.1.0)
  - dashboard: 7.8.0 (installed 7.0.0)
  - encryption: 2.16.0
  - federation: 1.18.0 (installed 1.10.1)
  - files_versions: 1.21.0 (installed 1.13.0)
  - firstrunwizard: 2.17.0 (installed 2.13.0)
  - nextcloud_announcements: 1.17.0 (installed 1.9.0)
  - recommendations: 2.0.0 (installed 1.3.0)
  - related_resources: 1.3.0 (installed 1.3.0)
  - support: 1.11.1 (installed 1.3.0)
  - suspicious_login: 6.0.0 (installed 4.3.0)
  - systemtags: 1.18.0 (installed 1.10.0)
  - text: 3.9.1 (installed 3.5.1)
  - user_ldap: 1.19.0
  - user_status: 1.8.1 (installed 1.0.1)
  - weather_status: 1.8.0 (installed 1.0.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

N/A

Additional info

I'm not sure if this should be considered a bug or enhancement. On the one hand, preview images are being generated but on the other hand, they're not being generated accurately to reflect the video.

[invario]

Have been researching the proper parameters to pass to FFMPEG to generate previews properly. Came across this link:
https://superuser.com/questions/1732301/convert-10bit-hdr-video-to-8bit-frames/1732304
It suggests that this parameter passed in would generate a proper preview/thumbnail for a HDR video. I tested it manually by generating a single frame and it appears to work. Interested in discussing this with anyone, thanks!
-vf zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p

edit: AAAAAND, boom goes the dynamite. zscale isn't on every ffmpeg version so this won't work for everyone. Hmm...

edit2: Wow this is turning out to be more complicated than I thought. So FFMPEG needs zscale but not every FFMPEG is compiled with zscale enabled. Then, even with zscale, you can not use the same parameters for HDR source video as you would for SDR source video because if you do, the resultant image/preview file is very, very dark and generated wrong. So in order to properly do this, the source video must be read, and then if, and only if it is HDR should the proper parameters get added to the command line for FFMPEG. Maybe a config.php parameters needs to be added to indicate if the FFMPEG installation has zscale available, otherwise default to no?

[invario]

I've been working on this a bit and have figured out things somewhat but am taking/soliciting input before I start hacking things on my end so that effort/work isn't wasted. Key notes to takeaway:

• ffmpeg binary with zimg compiled is required in order to use the zscale filter. As such, a configuration parameter to enable this should be added. preview_ffmpeg_enable_hdr? Default to 0, disabled since some platforms do not come with zimg compiled in ffmpeg. FreeBSD, for example, I had to build ffmpeg from scratch unfortunately.
• ffprobe (which is packaged with ffmpeg) is also required
• The ffmpeg binary path is located by https://github.com/nextcloud/server/blob/master/lib/private/PreviewManager.php#L380, and there is also a configuration parameter (to manually set the path. Maybe add an equivalent configuration parameter for the ffprobe binary. preview_ffprobe_path?
• ffprobe command line to detect HDR is: ffprobe -v quiet -select_streams v:0 -show_entries stream=color_transfer -of default=noprint_wrappers=1:nokey=1 HDRvideo.mp4
• if the value returned by ffprobe is "smpte2084" or "arib-std-b67", this indicates the movie is HDR. Otherwise, it is SDR
• if, and only if, the movie is detected as HDR by ffprobe, extra command line switches should be added to ffmpeg -vf zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p Otherwise, generate preview as normal
• in all cases, fallback to existing preview generation mechanism

Thoughts, comments, suggestions?

[invario]

Okay, I've done testing locally and it appears to be working on my end. I contained all the changes to Movie.php only since there is a default/fallback to the existing (and working) ffmpeg preview generation that already performs its own checks.