Not all mpls files are shown

[hubblec4]

Hi Rocky

When I load(with the GUI) a Blu-ray, for example Terminator 2, than not all mpls are shown.

I guess you use a filter to omit some mpls files.
Could you explain on what DGDemux filters.

Is there a way to get all mpls files?

[Rocky]

DGDemux just enumerates the .mpls files in the PLAYLIST directory and shows all of them, i.e., there is no filtering.

How are you determining that not all are shown? How many are there in total and how many are missing?

After you open the disk, don't click on an MPLS, and leave the GUI open. Then inspect the demuxXXXX.txt file in the same directory as DGDemuxGUI.exe, where XXXX is the PID shown in the GUI to the left of the Abort button. Compare the list in that file to a directory listing of PLAYLIST. If they match, then it is a GUI problem and we will go from there.

[Rocky]

After you open the disk, don't click on an MPLS, and leave the GUI open. Then inspect the demuxXXXX.txt file in the same directory as DGDemuxGUI.exe, where XXXX is the PID shown in the GUI to the left of the Abort button. Compare the list in that file to a directory listing of PLAYLIST. If they match, then it is a GUI problem and we will go from there.

Hmm, I just did that for CARS_2 and they don't match. Investigating... Ha ha, something to do today.

[Rocky]

I had a check that was intended to ignore "weird" MPLS files. By weird I mean those with the same M2TS file listed one after another. E.g., here is the list for CARS_2 00020.mpls:

00040.m2ts+00040.m2ts+00040.m2ts+00040.m2ts+ ...

Can't remember but somebody probably asked for that check. Don't ask me to make it optional!

Are all of your missing MPLS files like that? Do we really want to keep them? I suppose keeping them isn't a problem if we really value showing all MPLS files. OTOH, these weird playlists appear to be part of the studio's obfuscation strategy, and their presence in the list complicates identifying the main playlist.

[hubblec4]

Yes, the missing mpls have the same m2ts multiple times.
Good to know that you have a filter there and it is not a fault.

And I can also understand why there was a request to ignore such mpls files. The demuxed content is than hugh.

BUT
this mpls files are mostly used for the main menu screen.

cE has an option to ignore m2ts files which are repeated.
You have to know that such mpls files often also contain two m2ts files.
The first m2ts file is used only one time but the second is used multiple times.

It makes no sense to demux from such mpls the whole content.
MKVToolNix(MTX) uses also all m2ts file from the mpls list, so I have to feed MTX with the m2ts files,
or I demux all streams with cE which feeds eac3to only with one (or two) m2ts files.

With cE and Matroska ordered chapters I get a final mkv with only one m2ts but it plays so long like the mpls duration.

Should I open a new feature request "Remove duplicated m2ts from the m2ts list"?

[Rocky]

Should I open a new feature request "Remove duplicated m2ts from the m2ts list"?

Sorry, I don't follow your explanation. Please explain like you're talking to a five-year-old what the purpose of this option is, and the consequences of not having it. Why do you even need these playlists? I am disinclined to do this absent a really good justification. I'm also very stingy with new options. So if you can handle it within cE that will be preferred for us. You say this:

"cE has an option to ignore m2ts files which are repeated."

Why isn't that enough?

You know me, my friend hubblec4. I don't like to implement things whose purpose I do not understand.

[hubblec4]

Sorry, I don't follow your explanation. Please explain like you're talking to a five-year-old what the purpose of this option is, and the consequences of not having it.

Yes, no problem I will try to explain in more details

Why do you even need these playlists?

I keep these menu videos sometimes to have a menu in an mkv as well.
Especially with series, I find these menu videos very chic and like to keep them.
I plan to expand my cE->BD2mkv editor to create a single mkv of an entire Blu-ray, which preserves all videos
and with an interactive Matroska menu to navigate through the content.

I am disinclined to do this absent a really good justification. I'm also very stingy with new options. So if you can handle it within cE that will be preferred for us. You say this:

"cE has an option to ignore m2ts files which are repeated."

Why isn't that enough?

cE can feed eac3to with a free random m2ts list, but not DGDemux.
cE scans such mpls files creates the m2ts list and remove all duplicates.
This abbreviated m2ts list is used as input for eac3to and also for MKVToolNix.

OK, if there is only one m2ts which is repeated, this is simple, the m2ts list has only one m2ts after removing duplicates and MKVToolNix handles all fine.
But with the other case where two m2ts files are used, I like to demux the streams because to perform Audio gap-handling.

I know DGDemux will never have a free-m2ts-list-input feature and this is fully fine to me, but cE is not able to control DGDemux to ignore the duplicated m2ts files.

You know me, my friend hubblec4. I don't like to implement things whose purpose I do not understand.

That's perfectly understandable.

[Rocky]

Thank you. Thinking...

[hubblec4]

Some thoughts:
cE offers an option to remove the duplicates, but I could remove this and do the duplicate removal always by default.
Because it is nonsense to append the same content again and again.

This means DGDemux could do this also by default and so is no new switch necessary.

[Rocky]

Yes, I was thinking that too. I'll do the duplicate removal for you but I'm undecided still whether to have an option. Maybe I'll make it unconditional for now and add an option in the future if it ever becomes problematic for some use case. I mean after all, we used to throw out the whole MPLS, so how bad could it be?

[Rocky]

A question:

If I remove the duplicates, do you want the playtime returned as the original full time or the playtime with the duplicates removed? I'm tending towards returning the truncated playtime.

I temporarily removed build 70 because I had uploaded the wrong DGDemux.exe.

EDIT: build 70 was restored.

Be warned this duplicate removal change will need a few days as it is highly nontrivial due to interaction with -sf, -sl, -sk, and -ep.

[hubblec4]

... I'll do the duplicate removal for you but I'm undecided still whether to have an option.

Many thanks.
I'm also undecided for the moment. Already is everything working with this option.
I guess not much users are interested in this mpls files so I think an automatic remove is fully fine.
cE set the remove option also to "yes" automatically if there are duplicatem2ts files and a user have to uncheck the option to keep all m2ts fils.

Maybe I'll make it unconditional for now and add an option in the future if it ever becomes problematic for some use case. I mean after all, we used to throw out the whole MPLS, so how bad could it be?

Yes, sounds a good idea and no I think it could not be bad.

A question:

If I remove the duplicates, do you want the playtime returned as the original full time or the playtime with the duplicates removed? I'm tending towards returning the truncated playtime.

For the mpls list in DGDemux I think the original mpls duration is better.
Keep in my that also the chapters not longer entirely correct. To cut the chapters makes in this case sense, for a normal user.

For cE, I need only the demuxed streams, all time relevant thing and chapters are handled by cE already.

[Rocky]

Yeah, I knew about chapters too.

Let me ask you frankly... How bad would it be if I continued to ignore these weird playlists? Is it a deal-breaker?Honestly, this is going to be a difficult change for me.

[hubblec4]

Let me ask you frankly... How bad would it be if I continued to ignore these weird playlists? Honestly, this is going to be a difficult change for me.

That would be a bit of a shame, of course, but not the end of the world either. (I hope this sentence makes sense in English, translated by google)
If necessary, I will simply stop demuxing for such mpls files and hope that MKVToolNix can process everything properly.

But what is when you ignore all the other switches -sf, -sl...and so on.
All this can not be used, because we have mostly only one remain m2ts file.
If two m2ts files remains it makes also no sense to remove the first or last m2ts.

[Rocky]

I could implement the dup removal with an option, and if it's enabled then I disable -sf/-sl/-sk/-ep. That would be not so difficult. I'll do some experimenting.

I hope this sentence makes sense in English

Sounds like perfect colloquial English. Translation has come a long ways from the early days.

[hubblec4]

I could implement the dup removal with an option, and if it's enabled then I disable -sf/-sl/-sk/-ep. That would be not so difficult. I'll do some experimenting.

Wow, many many thanks and that's exactly what I mean, you are willing to find solutions, that's so great!!

[Rocky]

Hehe, thanks. I can argue that it's a special for cE, and so if you enable it as a normal user, it may not be what you expect.

[Bullwinkle]

Such a breath of fresh air to huddle with such a fine hoomin as hubblec4. Not to say there's much hope for hoominity, but as they say, think global but act local.

[Curly]

Huh? I can't see it.

[Rocky]

@hubblec4

Can you test this build please? Changes are:

1. All MPLS files are detected, including the weird ones.
2. There is a new option '-collapse' for DGDemux that will remove duplicates from the M2TS list.

Notes:

a. It's up to you to not enable -sf/-sl/-sk/-ep when you give -collapse.
b. I may or may not expose -collapse in the GUI. As it's a special for cE, arguably
there's no need for GUI support. Your thoughts?
c. Duplicates must be adjacent to be removed. E.g., if the list is:

00000.m2ts
00001.m2ts
00000.m2ts

then nothing is removed. OTOH if it is:

00000.m2ts
00000.m2ts
00001.m2ts

then one of the 00000.m2ts files is removed.

https://rationalqm.us/misc/DGDemux_hubblec4.rar

Sample:

DGDemux -l "D:\tmp\demux0000.txt" -i "G:\Rips\CARS_2\BDMV\PLAYLIST\00020.mpls" -thd 1 -collapse -o "D:\tmp\DGDemuxGUI test\0000" -demux 1011,1100,Chapters

[hubblec4]

Can you test this build please? Changes are:

Yes I will do and report my results. But it won't be for an hour.

a. It's up to you to not enable -sf/-sl/-sk/-ep when you give -collapse.

OK, and good to know. Only for interest, if I use one of the switches should I expect issues?

b. I may or may not expose -collapse in the GUI. As it's a special for cE, arguably
there's no need for GUI support. Your thoughts?

Mmh good question. I'm sure this feature is not often used by normal users,
therefore I would say you can omit the support for the GUI until a user asks for this.
But on the other hand it could be useful for quicker test for developing.
I hope it's OK if I let you decide.

c. Duplicates must be adjacent to be removed. E.g., if the list is:

00000.m2ts
00001.m2ts
00000.m2ts

then nothing is removed. OTOH if it is:

00000.m2ts
00000.m2ts
00001.m2ts

then one of the 00000.m2ts files is removed.

OK, and I think this is a good solution.
I have never seen the first case where the same m2ts are not in a row.

What means OTOH?

A little note on the "weird" mpls:
This isn't actually weird, it's actually clever.
Instead of packing a 5 hour menu video with 5gb or more onto the disc, you simply repeat the content of an m2ts until you get to 5 hours.

The Matroska ordered chapters follow the same principle.
You can also create a 5h video with just one frame.

[Rocky]

Yes I will do and report my results. But it won't be for an hour.

No problem, thank you. Take your time. I'll be taking my afternoon nap.

OK, and good to know. Only for interest, if I use one of the switches should I expect issues?

Don't know, haven't thought it through.

I hope it's OK if I let you decide.

Sure, no problem.

What means OTOH?

On the other hand.

[hubblec4]

It was easier to implement as I thought.
It works perfect.
I used for a test an mpls with a 10h duration and the demuxed video is reduced to 3min.

You saved me a lot of time with this feature and I had to write only two lines of code.

Again many thanks for all your work.

[Rocky]

Awesome. You are most welcome and thank you for your testing.

I'll wait a little while for us to get some more experience with things before releasing it. And I'm still oscillating between implementing it in the GUI and not. Seems to be a damped oscillation, thankfully.

20 lines of code on my side, BTW. It still took several hours, though, because it was quite tricky to do the deletions correctly and robustly, including all the corner cases.

[hubblec4]

...And I'm still oscillating between implementing it in the GUI and not. Seems to be a damped oscillation, thankfully.

The GUI presents now also this special mpls files and so it would be good to have the collapse option also in GUI.

20 lines of code on my side, BTW. It still took several hours, though, because it was quite tricky to do the deletions correctly and robustly, including all the corner cases.

Yes, the code is mostly fast written but the research like you said takes a lot of time sometimes.
I know exactly what you mean