Auto indexing for DGDecode

[Rocky]

That was interesting. Re-download the 2009 build and update DGDecode.dll. Lemme know if it works or not.

Also an improvement request if I may: could we have a string argument to set the DGIndex configuration. Reason: a fresh DGIndex.exe at default settings will demux all audio tracks and create a bunch of files alongside the source - I don't want to have to rely on instructing the user to manually open DGIndex.exe themselves and set it up to only create the d2v file otherwise they will wonder why its dumping all these audio files and consuming disk space.

We do the following in DGDecodeNV. Is that what you want in DGDecode?

* When opening and indexing a media file directly with DGSource(), by default streams are
now not demuxed. If you want to have them be demuxed while indexing, add an @ character to the
DGSource() filename, e.g., DGSource("file.ts@").

Especially could we please have the -OF parameter so we can specify where to put the d2v file. I'm dealing with a huge library and need to have control over where my files go.

How do we signal that to DGSource()? Thinking...

EDIT: I guess we add a new DGSource() parameter 'indexing_path' specifying the directory to use when auto-indexing. Is that OK?

EDIT: Are you using 32-bit or 64-bit?

[flossy_cake]

That was interesting. Re-download the 2009 build and update DGDecode.dll. Lemme know if it works or not.

Also an improvement request if I may: could we have a string argument to set the DGIndex configuration. Reason: a fresh DGIndex.exe at default settings will demux all audio tracks and create a bunch of files alongside the source - I don't want to have to rely on instructing the user to manually open DGIndex.exe themselves and set it up to only create the d2v file otherwise they will wonder why its dumping all these audio files and consuming disk space.

We do the following in DGDecodeNV. Is that what you want in DGDecode?

* When opening and indexing a media file directly with DGSource(), by default streams are
now not demuxed. If you want to have them be demuxed while indexing, add an @ character to the
DGSource() filename, e.g., DGSource("file.ts@").

Especially could we please have the -OF parameter so we can specify where to put the d2v file. I'm dealing with a huge library and need to have control over where my files go.

How do we signal that to DGSource()? Thinking...

EDIT: I guess we add a new DGSource() parameter 'indexing_path' specifying the directory to use when auto-indexing. Is that OK?

EDIT: Are you using 32-bit or 64-bit?

I realised I can just provide my own .ini file alongside DGIndex.exe, so no need to implement a DGIndex config string.

The custom path for .d2v files would still be handy - if we could do something like

Code: Select all

mpeg2source("C:\MyVob.vob", indexing_path="C:\MyD2Vs")
DGSource("C:\MyVob.vob", indexing_path="C:\MyDGIs")

The 2009 build is good now regarding the variable getting overwritten, thanks

edit: I'm using x64 for mpeg2source and DGSource

[Rocky]

OK, thank you. I'll try to find time to do that fairly soon.

[Rocky]

Give this a try for DGDecode (64) (avisynth). You must have a trailing \ on the indexing path.

MPEG2Source("lainvob.vob", indexing_path="d:\tmp\")

https://rationalqm.us/misc/DGDecode_flossy_cake.rar

Let me know if it works for you. If it's good I'll add Vapoursynth and port to DGDecodeNV(). I'm thinking of ditching 32-bit DGMPGDec. I also have to update the manual.

[flossy_cake]

Give this a try for DGDecode (64) (avisynth). You must have a trailing \ on the indexing path.

MPEG2Source("lainvob.vob", indexing_path="d:\tmp\")

https://rationalqm.us/misc/DGDecode_flossy_cake.rar

Let me know if it works for you. If it's good I'll add Vapoursynth and port to DGDecodeNV(). I'm thinking of ditching 32-bit DGMPGDec. I also have to update the manual.

Working for me, except for the .log file which is still created alongside the source file.

[Rocky]

The log file is created by DGIndex so it can't be controlled by the indexing_path parameter.

I'll make a release.

[flossy_cake]

Sorry to keep making all these demands but I just got The Simpsons Region 4 PAL DVD and S01E01 plays at double rate (50fps) with repeat field flags obeyed - ignoring the flags plays at correct speed 25fps. So I must ignore repeat field flags for this particular DVD. I get the same result with LWLibAV and DGSource so I don't think it's a decoding error, it must just be the DVD was mastered incorrectly somehow. Therefore would it be possible to add another argument for us to select whether we want to obey or ignore repeat field flags?

[Rocky]

I'd sure like to see that clip so if you could link it that would be appreciated. Just the first 50MB of the first VOB would be dandly. Use DGSplit to cut it.

Generally, ignoring pulldown is dangerous without looking carefully at the source clip's mix of hard vs. soft pulldown, etc. You could burn your house down or put out an eye. Nevertheless, since you are willing to edit the INI file you can also edit the D2V file's Field_Operation= variable to get what you want.

[flossy_cake]

Oh it looks like DGIndexNV doesn't support repeat field flags, or at least I can't see any option for it in the GUI But when I play my mkv DVD remux MPEG2's through it, I get 30i instead of 24p so it seems to be behaving as if it is obeying repeat fields.

Well, now I'm confused

Nevertheless, since you are willing to edit the INI file you can also edit the D2V file's Field_Operation= variable to get what you want.

Avisynth doesn't have any internal functions to edit INI files - I'm merely including my single prebaked INI file alongside the DGIndex.exe together with all other dependencies of my script.

[flossy_cake]

The log file is created by DGIndex so it can't be controlled by the indexing_path parameter.

In that case I tried to disable it with Enable_Info_Log=0 in DGIndexNV.ini but it still gets created (not an issue for MpegSource2 though).

[flossy_cake]

Is there no x64 dll for dgmpgdec3000.zip?

[flossy_cake]

And could we please have indexing_path for DGSource as well

[Rocky]

Oh it looks like DGIndexNV doesn't support repeat field flags, or at least I can't see any option for it in the GUI But when I play my mkv DVD remux MPEG2's through it, I get 30i instead of 24p so it seems to be behaving as if it is obeying repeat fields.

RTFMs! DGMPGDec uses a setting in the GUI that is stored in the D2V file. DGDecNV uses the parameter "fieldop" to DGSource().

Avisynth doesn't have any internal functions to edit INI files - I'm merely including my single prebaked INI file alongside the DGIndex.exe together with all other dependencies of my script.

That's true. It's a manual thing, because there is no reliable way to automatically determine the correct fieldop to use for a given clip. I won't enable you to shoot yourself in the foot.

In that case I tried to disable it with Enable_Info_Log=0 in DGIndexNV.ini but it still gets created (not an issue for MpegSource2 though).

Hard to believe but I'll look into it.

Is there no x64 dll for dgmpgdec3000.zip?

It's right there in the distribution. It's called DGDecode.dll. 32-bit is no longer supported.

I requested the clip sample and you just ignored that. So I'll ignore any further requests from you as long as you continue to ignore me. Send a PM with a link if it's not something you want to share with everybody.

[flossy_cake]

RTFMs!

lol I did read the manual just the wrong one - DGIndexNVManual.html. Anyway I think it's a fair inference if the setting is present in one GUI but not the other then the second one probably doesn't support that setting. I tried to use critical thinking to figure out why this might be the case and I thought to myself it must be because the NVidia driver is doing the MPEG2 decoding and the NVidia driver probably doesn't implement repeat flags.

It's a manual thing, because there is no reliable way to automatically determine the correct fieldop to use for a given clip. I won't enable you to shoot yourself in the foot.

Well yes but I can already shoot myself in the foot by setting it to disobey in the DGIndex GUI. But I would never disobey them, only as a last resort if some file is not playing nice with it.

It's right there in the distribution. It's called DGDecode.dll. 32-bit is no longer supported.

Well visual studio always puts it in a separate x64 folder and that's how it's been in your previous releases and other Avisyth filters use that convention so I was just making an inference to best possible explanation. iirc there is a way to tell if a dll is 32/64 but I forgot what it was.

I requested the clip sample and you just ignored that.

I made a clip and uploaded it to gdrive but then shortly after figured out it's not a repeat flag issue so I edited my reply, sorry if that seemed like I was ignoring you.

Anyway everything is great and working nicely but I use DGSource 99% of the time and just need that indexing_path setting and then you won't have to hear from me probably ever again

[Rocky]

Anyway I think it's a fair inference if the setting is present in one GUI but not the other then the second one probably doesn't support that setting.

That inference is correct, the GUI does not support it, but the DLL does.

Well visual studio always puts it in a separate x64 folder and that's how it's been in your previous releases and other Avisyth filters use that convention so I was just making an inference to best possible explanation. iirc there is a way to tell if a dll is 32/64 but I forgot what it was.

Things changed because 32-bit was dropped. Make a script with the DLL and open in 64-bit VirtualDub. If it succeeds then the DLL is 64-bit. If not, then32-bit.

Anyway everything is great and working nicely but I use DGSource 99% of the time and just need that indexing_path setting and then you won't have to hear from me probably ever again

It's on the plate, don't worry. And I love hearing from you.

[flossy_cake]

indexing_path:
If this option is missing or is given as "" then the folder containing the source file will be used.

When indexing_path is not specified, I'm observing it placing the files alongside the .avs file whereas DGSource places them alongside the source file.

[Rocky]

The log file is created by DGIndex so it can't be controlled by the indexing_path parameter.

In that case I tried to disable it with Enable_Info_Log=0 in DGIndexNV.ini but it still gets created (not an issue for MpegSource2 though).

I checked the code. The log is always generated for a CLI invocation. Can't remember why it's that way but probably someone asked for it at some point. Can't see any reason to change it.

Almost done with indexing_path for DGSource(). It's all coded but needs tasting.

[flossy_cake]

Much appreciated thanks.

[Rocky]

You are welcome. Released as build 255. Let me know if I screwed up anything.

Pretty soon the build number will have to go to 16 bits.

[flossy_cake]

Thank you, just a couple of issues to report

1. The first argument is overwritten by DGSource like it was with mpeg2source

2. When mpeg2source(indexing_path=undefined) the indexing related files are created alongside the .avs script file whereas DGSource, lwlibav & ffms2 create them alongside the source clip. If mpeg2source could follow the same convention that would make life a lot easier as my scripts involve use case scenarios where indexing_path=undefined which means for example after AudioDub(mpeg2source, lwlibaudiosource) I end up with some indexing files going to one location and others to another location.

________________________________________

On a totally separate note I'm finding DGSource can sometimes have audio sync issues when multithreading is enabled in Avisynth. I realise Avisynth multithreading is dodgy and trying to chase down the cause is probably something you're not interested in doing, but I thought I'd ask in case there was some parameter that could be passed to the NVidia driver to somehow maintain sync.

I can also do NVidia decoding with lwlibavvideosource(prefer_hw=1) and it's definitely using the Nvidia GPU according to GPU-Z's "video engine load" statistic, and so far it seems to be ok with Avisynth multithreading. Anyway this isn't something I'm asking you to put time into but just thought I'd mention it on the off chance there is some simple parameter tweak that could resolve it which I'm not aware of.

Thanks

[Rocky]

I'll fix 1 and 2 today and make slipstreams.

Regarding the audio sync can you give me the script and if possible a stream that I can use to duplicate it? Also the more you can explain the better. How are you determining there is a sync problem, etc.? Your full process.

EDIT: Number 1 fixed and slipstreamed.

[Rocky]

2. When mpeg2source(indexing_path=undefined) the indexing related files are created alongside the .avs script file whereas DGSource, lwlibav & ffms2 create them alongside the source clip.

My testing shows them behaving the same. Are you using a full path? If your script has a full path:

MPEG2Source("g:\streams\mpg\lain.vob")

then the index file will be created in the directory of the path, i.e., in the source file directory.

If your script does not have a full path:

MPEG2Source("lain.vob")

then the index file will be created in the current directory, i.e., in the script file directory.

If you aren't getting this behavior or you think it is still a problem, please provide a script and description of the scenario and how it fails for you.

[flossy_cake]

If you aren't getting this behavior or you think it is still a problem, please provide a script and description of the scenario and how it fails for you.

Code: Select all

file = "C:\DVDs\disc1.vob"

LWLibavVideoSource(file)  # creates C:\DVDs\disc1.lwi
FFVideoSource(file)       # creates C:\DVDs\disc1.ffindex
DGSource(file)            # creates C:\DVDs\disc1.dgi
Mpeg2Source(file)         # creates <ScriptDir>\disc1.d2v

Regarding the audio sync can you give me the script and if possible a stream that I can use to duplicate it? Also the more you can explain the better. How are you determining there is a sync problem, etc.? Your full process.

I'm not yet able to produce for you a small isolated test example as the issue only occurs under certain scenarios in Avisynth which I've yet to determine. I will have to spend a few hours turning things on and off one at a time to see what filters are provoking it. For instance if I remove all Text() overlays then it almost clears the issue (but not quite) and if I bypass one of my ScriptClips then that seems to resolve it completely. Probably an issue with Avisynth as it's known to be dodgy with multithreading.

Here is a prerendered clip of what it looks like on my end. Notice how it resyncs itself a couple of times at the beginning (@ 1 second & 5 seconds) and then comes good. But it doesn't always do that, sometimes just remains out of sync until I force a manual seek during realtime playback. Turning off multithreading resolves it completely, that's why I think it's a multithreading issue.

edit: just noticed the frame numbers in the top right corner (previous_frame -> current_frame) is still increasing in a linear fashion at the time of the sync stutters. I'm not sure exactly what that implies, but it can't be good.

[Rocky]

Code: Select all

file = "C:\DVDs\disc1.vob"
...
Mpeg2Source(file)         # creates <ScriptDir>\disc1.d2v

That doesn't happen for me. Here is my script:

loadplugin("D:\Don\Programming\C++\DGMPGDec_VS2019\DGDecode64\x64\debug\DGDecode.dll")
file="G:\Streams\MPG\Program Streams\lain.vob"
MPEG2Source(file)

I simply drag and drop the script into VirtualDub2. The D2V is created in "G:\Streams\MPG\Program Streams". How are you invoking the script?

Here is a prerendered clip of what it looks like on my end.

What is a "prerendered clip". Are you seeing this when directly playing a script, or are you doing a full encode and then seeing it in the encoded file? This is why I asked you for your process.

But it doesn't always do that, sometimes just remains out of sync until I force a manual seek during realtime playback.

What is "during realtime playback"? Same point as above. You have to tell me what you are doing! And why can't you give me your script so I can see your multithreading setup, etc.?

[flossy_cake]

That doesn't happen for me.

Oh interesting, something else must be going on then. Are you absolutely sure? Like, did you delete G:\Streams\MPG\Program Streams\lain.d2v , then watch it get recreated and check the time stamp on the file that it wasn't an older version previously created (sorry if this question is insulting to your intelligence, but I got caught out by this a few times, just trying to be thorough). edit: is it possible your .avs script is also in G:\Streams\MPG\Program Streams?

What is a "prerendered clip".

An Avisynth script rendered to a video file using a video codec such as h.264 - I used ffmpeg.exe at the command line to make that demo.mkv. Versus playing an .avs file directly in a media player in my case MPC-HC (or AvsPMod, VirtualDub etc.) aka "realtime usage".

Are you seeing this when directly playing a script, or are you doing a full encode and then seeing it in the encoded file?

Both, but I can't come up with a script demo for you to try on your system because currently I can only reproduce it in my Avisynth script which is quite large and I'm not ready to release it yet, and you would just look at it and have no idea what's going on nor would you be interested in reading through my large script and trying to debug it, so I need to come up with a very short and simple example for you to reproduce it, so far I cannot do that.