Sunday, November 16, 2014

MythTV: Remove commercials and transcode to H.264

If you use MythTV and would like to automatically mark/remove commercials and transcode your recordings to H.264 (Mpeg4),  you should add this script as a "user job" in your mythtv-backend setup.

Pre-requisites:

  • A working MythTV backend setup
  • HandBrakeCLI installed.  
    • To install on Ubuntu, try:  sudo apt-get install handbrake-cli
  • Your MythTV database username and password:
    • You can find these under /etc/mythtv/config.xml
    • Add these to the DBUSER and DBPASSWD lines in the script.  Keep quotes.


Script functions:

0. Invoked as a user-job after a MythTV recording completes
1. Query database to check if invoked for a commercial-free channel
2. If commercial-free channel, go to step 7
3. Flag commercials using "All" methods, copy flagged commercials to cutlist
4. Lossless transcode to remove commercials from the file
5. Rebuild the seek table. Clear cutlist
6. Update database to point to recording with no commercials
7. Remove stale bookmarking and seek information from the database
8. Use Handbrake to transcode recording to H.264 (MPEG-4):
    -- preserve original audio track (5.1, dts etc.)
    -- preserve subtitles
    -- Use x264 encoder for HD-quality video with much reduced file-size
    -- deinterlace video if necessary
    -- create .mp4 file format compatible with Roku or Apple/Android devices
 9. Update database to point to the new .mp4 transcoded file
10. Rebuild seektable for new .mp4 transcoded file


Download and install:

0. You can download the script here.
1. Change the DBUSER and DBPASSWD lines in the script to match your settings.
2. Save the script (as root) to /usr/local/bin
3. Invoke mythtv-setup and navigate to the user-jobs section.
4. Provide the 4 arguments required for the script: %DIR% %FILE% %CHANID% %STARTTIMEUTC% as shown below:


5. For a new recording schedule via Mythweb, select the name of the user job you just added:


6. You should see the job launch after the recording with a "T" icon under "Recorded Programs" in Mythweb, indicating that the file has been transcoded:


34 comments:

  1. This isn't working anymore for Mythtv .27 (mythbuntu). It keeps trying to remove commercials from a zero-byte tmp file.

    ReplyDelete
  2. The problem was that the content was already H.264 due to being recorded from Hauppauge HD-PVR, so all I really needed was to cut the commercials out. That's not something mythtranscode can do.

    ReplyDelete
  3. Yep, if you're already recording h.264 content, then all you need is to modify the script to go up to step 7 above. That is, don't perform the HandbrakeCLi transcode.

    ReplyDelete
  4. This is an awesome script and it's working really well - thanks!

    Is there anyway that it could do a rename at the end to Show Name, Season, Episode like : "Person of Interest - S01E22 - No Good Deed.mp4" Then it would be dead simple to import to something like Plex.

    Thanks for the awesome script!

    ReplyDelete
  5. Works flawlessly no issues.

    @Keith Look up mythicalLibrarian to do what you are looking for.

    ReplyDelete
    Replies
    1. @Chris

      How do you get these two user jobs to play nicely? Which one do you run first? MythicalLibrarian or this Handbrake transcode job? Do you allow Mythical to move the files or do you let this transcode job move them?

      Delete
  6. If the user job fails, where I can see the output log?

    ReplyDelete
  7. Under /tmp, you should see a directory called postProcess-PID, where PID is the process ID of the job that failed. In that directory, you'll find some logfiles, including a file called ppr.log.

    Note that if the job passes successfully, this directory is removed.

    ReplyDelete
  8. anyway to get around the call to ionice on OS X?

    ReplyDelete
    Replies
    1. Not entirely sure, but nice/renice should have I/O scheduling capabilities in MacOS, so you can try that instead.

      Delete
    2. I'm getting the same behavior and I've tried running the script various ways. i.e. with and without the parameters and using full paths to both the sh and mpg files. Any ideas?

      Delete
  9. This job isn't working for me and there are no logs in /tmp...

    ReplyDelete
    Replies
    1. Can you try running the job on the command-line and check where it fails?

      Delete
    2. How do I run it from the command line? I need to pass it some arguments, right?

      Delete
    3. I upgraded my MythTV system, and now your job is "working." However, it seems to never end...

      Delete
    4. What can you glean from the log file? Which step is it hung on?

      Under /tmp, you should see a directory called postProcess-PID, where PID is the process ID of the job that failed. In that directory, you'll find some logfiles, including a file called ppr.log.

      Delete
  10. I keep getting an error: "ERROR: Invalid audio codec: 0x100" and I have not figured out how to fix the issue with Handbrake. I wonder if @trnsmatrix is having the same issue. I have tried a few different -e options with no success I wonder if I need to install something. I am running on mythbuntu.

    ReplyDelete
  11. Perhaps you're missing the AAC codec, since handbrake tries to convert the second audio track to AAC for playback on mobile/roku devices? I haven't seen this error before, but that would be my guess. I'd try:
    % sudo apt-get install ubuntu-restricted-extras

    May be worth a shot, or google for installing AAC support in case you don't have it.

    ReplyDelete
    Replies
    1. Got it figured out. It's an issue with Ubuntu 14.04. I used this link to get things working: http://ubuntuhandbook.org/index.php/2014/04/install-handbrake-ubuntu-14-04-trusty/

      Delete
  12. Got this script working great. However it cuts off the ending before even the credits roll. Anyone know how to tweak this so it doesnt cut off thw endings of shows?

    ReplyDelete
    Replies
    1. I haven't seen this happen, but my guess is that the commercial flagging is being too aggressive and cuts off the credits at the tail end of the recording. If so, you can try tweaking the argument to --method passed to mythcommflag in step 3, which currently uses "all" methods.


      Commflagging Options:
      --method Commercial flagging method[s] to employ:
      off, blank, scene, blankscene, logo, all, d2,
      d2_logo, d2_blank, d2_scene, d2_all

      Delete
  13. when I run this I get the following in the ppr.log

    "/usr/local/bin/removecom: File does not exist: GetPlaybackURL/UNABLE/TO/FIND/LOCAL/FILE/ON/mythtv/1232_20160203190000.mpg"

    Mythweb however reports the job completed successfully . Unfortunately, after the job is complete the file appears to be completely gone. I can not find it anywhere on the drive.

    Any suggestions?

    ReplyDelete
    Replies
    1. Did you add the 4 arguments to the script as required and stated above in mythtv-setup in steps 3 and 4?

      Does it work when you run the script standalone?

      Delete
  14. Hi Patna

    Great instructions thank you and a nice script. I'm close to getting your script to work for me but it's failed each time so far. Mythweb reports:

    Status: Errored
    Comments: Error User Job returned non-zero, check logs.

    The following is one of my (ppr.log) error log files:

    /usr/local/bin/postProcessRecording.sh: Generating cutlist for 1005_20160313095500.mpg.../usr/local/bin/postProcessRecording.sh: Removing commercials...
    Replaced 1005_20160313095500.mpg.tmp with 1005_20160313095500.mpg. Rebuilding seektable...
    Transcoding 1005_20160313095500.mpg to 1005_20160313095500.mp4
    HandBrakeCLI: unrecognized option '--no-opencl'
    unknown option (--no-opencl)
    Handbrake transcode failed with error: 1

    I've also installed the Handbrake GUI, for testing purposes, and have used it to successfully transcode. However, it presented only options to output to MKV an and not MP4.

    Is there maybe a tweak, I will need to make, to your script to get it working?

    Thanks

    Simon

    ReplyDelete
    Replies
    1. I removed the --no-opencl flag from the list of arguments since someone else had also mentioned that they were having this problem. Please re-download the script.

      The "--format mp4" in HandbrakeCLI will produce an mp4 file. This works. Try running mediainfo on your output video file.

      Delete
  15. Hi Patna

    Thanks for your very quick reply. It did some further testing, as you suggested in an earlier post, by running HandbrakeCLI interactively from the command line.

    Simply removing --no-opencl from the command line list of arguments resulted in a different error for me:

    "ERROR: Invalid audio codec: 0x100" As reported by Ryan L above.

    I've since found that if I install Handbrake using:

    sudo add-apt-repository ppa:stebbins/handbrake-releases
    sudo apt-get update
    sudo apt-get install handbrake-gtk handbrake-cli

    Then the original command within your script worked just fine.

    I'll next try running your script from within MythTV and let you know.

    Thanks

    Simon

    ReplyDelete
  16. This queued and ran, but after a little while it says completed, but when I go to recordings the show has an X next to it and says file not found??

    ReplyDelete
  17. update needed for myth 0.28. Stopped working after update. Appears the db update after transcoding is no longer working. Left with a viewable *.mp4 file, but the db points to a *.ts file instead of the *.mp4.

    ReplyDelete
    Replies
    1. Think I have fixed it. Myth 0.28 saves and transcodes to .ts files instead of .mpg files. Edit the script and modify the OUTFILE line to match the following:

      OUTFILE=`echo "$2" | sed s/\.mpg/\.mp4/g | sed s/\.ts/\.mp4/g` # Change extension to .mp4

      This worked for me.

      Delete
  18. Has anyone been able to get this functioning in 0.28 beside Dave? I tried Dave's suggestion on the OUTFILE update and it still gives me the ERROR: User Job returned non-zero, check logs

    Also, where is the log file saved?

    ReplyDelete
  19. Updated for MythTV v0.28 with another minor edit for commercial flagging. Thanks Dave for pointing out that 0.28 onwards saves recorded files as a .ts extension.

    Log files will be in /tmp in case any part of the script fails.

    cheers!

    ReplyDelete
  20. This script is working great, except in 0.28 there is a new web interface and on the home page there is a preview of the last 10 recordings. Well after running the script it receives this error: Image not found: /Content/GetPreviewImage?RecordedId=1031&Height=371&Format=JPG

    This is the directory listing of a recording before I ran the script:
    -rw-r--r-- 1 mythtv mythtv 8299413220 Feb 28 21:30 1071_20170301020000.ts
    -rw-rw-rw- 1 mythtv mythtv 11018 Feb 28 22:37 1071_20170301020000.ts.-1.100x56.png
    -rw-rw-rw- 1 mythtv mythtv 4177 Mar 4 16:47 1071_20170301020000.ts.-1.160x-1.jpg
    -rw-rw-rw- 1 mythtv mythtv 42012 Mar 1 05:07 1071_20170301020000.ts.-1.-1x371.jpg
    -rw-rw-rw- 1 mythtv mythtv 1054704 Feb 28 22:29 1071_20170301020000.ts.png

    This is the transcoded listing after the script:
    -rw-r--r-- 1 mythtv mythtv 3141740392 Mar 6 07:39 1071_20170301020000.mp4
    -rw-rw-rw- 1 mythtv mythtv 12876 Mar 6 07:40 1071_20170301020000.mp4.-1.100x56.png
    -rw-rw-rw- 1 mythtv mythtv 51053 Mar 8 05:56 1071_20170301020000.mp4.-1.-1x371.jpg
    -rw-rw-rw- 1 mythtv mythtv 3516 Mar 8 05:59 1071_20170301020000.mp4.-1.-1x67.jpg
    -rw-rw-rw- 1 mythtv mythtv 1132554 Mar 6 07:40 1071_20170301020000.mp4.png

    Seems to have a file missing. I found a forum post that touches on this subject, but I'm not a programmer and have no idea if or what the solution might be. Could you take a look, thanks!

    https://lists.gt.net/mythtv/users/587450

    ReplyDelete
  21. Hello ... thanks for all you work on this script ... trying to get it to work on 0.29 but the job queues forever, do not see any obvious issues in the back-end logs ... please can you update for 0.29 ... Thx Vik

    ReplyDelete
  22. If anyone is still using this, a recent update to HandBrakeCLI made the following parameter invalid: "--no-opencl" If you don't remove this parameter from the HandBrakeCLI command, handbrake will bomb and as a result your recording will be deleted.

    ReplyDelete