Dup Plugin for Avisynth
(Version 1.4)

by Donald A. Graft

Introduction

This plugin for Avisynth implements a robust duplicate frame detector. It is initially intended as a test bed for a new frame differencing algorithm, but to make it a useful filter in its own right, a copy feature has been implemented, whereby a frame that is close enough to its predecessor to be considered a duplicate will be replaced by a copy of the predecessor. This is intended for use in clips that have a significant number of duplicate content frames, but which differ due to noise. Typically anime has many such duplicates. By replacing noisy duplicates with exact duplicates, a bitrate reduction can be achieved. It's a free lunch! ('Marc FD' first implemented this copying functionality.)

The filter currently requires YUY2 input. It is not yet optimized for speed. Optimizations will be included in a future version.

The justification for the filter is simple. Many duplicate detectors are incapable of adapting to changes in the luminance levels of the various scenes in a clip, and they are badly affected by noise. They have delicate absolute thresholds that must be set by the user. This filter attempts to solve all of these problems.

Dup uses a small window that it scans over the frame, so it is capable of seeing very small changes. The effect of noise is reduced by means of automatic cancellation through a summation process. The frame difference is normalized to the range 0-100% to make setting the threshold more intuitive.

Dup Function Syntax

Dup uses named parameters. That means you do not have to worry about the ordering of parameters and can simply refer to them by name and put them in any order in the list of parameters. If you omit a parameter it takes its default value. For example, if you want to run Dup with a duplication declaration level of 5% and with debug enabled, you can simply say:

Dup(threshold=5,debug=true)

Any combination and order of named parameters is allowed. Remember, however, that you should always include empty parentheses if you are not specifying any parameters.

You can also use normal positional syntax without the names if you prefer. The correct parameter order is:

Dup(threshold,show,copy,maxcopies,debug)

So you could just use:

Dup(5,false,true,10,false)

Changing Default Parameter Values

If you do not like the defaults as documented below, you can set your own standard defaults if you are using Avisynth version 2.05 or later. To override the defaults, first create an Avisynth plugins directory and register it. You register it by putting the following lines in a text file called 'plugin.reg', changing the path as appropriate, and then right clicking on the file's icon and selecting Merge.

REGEDIT4

[HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth]
"PluginDir"="d:\\avisynthplugins"

Next, create defaults files as required in the plugins directory. For example, to set the default copy=true for Dup(), make a file called Dup.def and put this line in it:

copy=true

You can list as many parameter assignments as you like, one per line. Those not specified assume the values given below. Of course, you can always override the defaults in your scripts when you invoke the functions. NOTE: The lines in the defaults file must not contain any spaces or tabs.

Following is the syntax for Dup (replace parameter_list with your comma-separated list of named parameters):

Dup(parameter_list)

threshold (0.0-100.0, default 5.0): This parameter defines the percentage change in the most different 32x32-pixel window that is enough to declare a frame a duplicate. The way to think about it is that if the threshold is (say) 7%, then if any 32x32 pixel area changes by 7% or more, it is not a duplicate, otherwise it is. Note: 3-5% is a good general value. Use the show option to display frame difference values to help in setting the threshold if you prefer something other than the default.

You should always set your threshold above the noise level of the clip. The amount above will depend on how many duplicates you want to generate, i.e., the desired bitrate reduction. To determine the noise level, examine the frames and look for those having duplicate content, albeit with noise differences. You may find that frames that are content duplicates show a metric of as high as a few percent (due to noise). Determine the highest metric that you get for the content duplicates. Then set your threshold above that level with as much margin as you want to achieve the desired bitrate reduction.

show (true/false, default false): When set to true, the 32x32 pixel area that is most different is delineated with a white box, and metrics are written in the upper left of the frame. If the box has an X through it, the frame was declared a duplicate according to the configured difference percentage (threshold). Note: When the copy option is enabled, copied frames will not have a box because they are identical to their predecessor and so there is no most-different area! The text information will tell you which frame it is a duplicate of.

copy (true/false, default false): If this option is set to true, declared duplicates will be replaced with an exact copy of their predecessor. If there is a string of duplicates, the first frame in the string is used for all copies.

maxcopies (integer, default 10): Determines the maximum number of pure copies that can be emitted due to duplicate declaration. However, if the input stream has more than the maximum number of real duplicates, then, of course, they will all be output. This limit just forces the current duplicate sequence to end; the new frame is passed through and the comparisons restart with the new fresh frame.

debug (true/false, default false): This parameter enables debug output to the DebugView utility. This utility can be downloaded from my web site as given below.

Copyright (C) 2002, Donald A. Graft, All Rights Reserved.
Feedback/inquiries to neuron2@attbi.com.

For updates and other filters/tools, visit my web site:
http://shelob.mordor.net/dgraft/