New special mpls structure
New special mpls structure
Hi Rocky
I have started the download for the Blu-ray and I think It is a good time to bring up this issue.
I will provide the disc chunk which I got, so you can have a first look over it.
The important mpls files are 00006 and 00005. Both mpls files uses the 00001.m2ts file.
Mpls 00005 has only one play_item and a duration of 01:34:31.999666666.
Mpls 00006 uses 20 times the 00001.m2ts as play_item but not in full way and has only a duration of 00:25:29.194355556.
In DGDemux GUI, I can activate the -Collapse switch and only one times the 00001.m2ts file is used, but also the entire m2ts file is used.
Without activate the -Collapse switch the output of the extracted streams is very big (400gb)
MTX handles this mpls file fine. The created MKV plays exactly 00:25:29.194. (This was the answer from the user).
cE don't handles this case correctly because I feed MTX always with the m2ts files and never with the mpls files(for several reasons).
I attach also the MTX output which I got. It says not much, but you can see that MTX append also the play_items 20 times, but it reads only the duration given by new IN- and OUT- times for the 00001.m2ts file.
My guess for DGDemux is, that when the new play_item is reached and it is the same as the previous play_item, the IN- and OUT- times are not updated?
I have to wait until the download is finish to make correct statements.
I have started the download for the Blu-ray and I think It is a good time to bring up this issue.
I will provide the disc chunk which I got, so you can have a first look over it.
The important mpls files are 00006 and 00005. Both mpls files uses the 00001.m2ts file.
Mpls 00005 has only one play_item and a duration of 01:34:31.999666666.
Mpls 00006 uses 20 times the 00001.m2ts as play_item but not in full way and has only a duration of 00:25:29.194355556.
In DGDemux GUI, I can activate the -Collapse switch and only one times the 00001.m2ts file is used, but also the entire m2ts file is used.
Without activate the -Collapse switch the output of the extracted streams is very big (400gb)
MTX handles this mpls file fine. The created MKV plays exactly 00:25:29.194. (This was the answer from the user).
cE don't handles this case correctly because I feed MTX always with the m2ts files and never with the mpls files(for several reasons).
I attach also the MTX output which I got. It says not much, but you can see that MTX append also the play_items 20 times, but it reads only the duration given by new IN- and OUT- times for the 00001.m2ts file.
My guess for DGDemux is, that when the new play_item is reached and it is the same as the previous play_item, the IN- and OUT- times are not updated?
I have to wait until the download is finish to make correct statements.
- Attachments
-
- output.txt
- (33.33 KiB) Downloaded 147 times
-
- MPLS-multiple-IN-OUT-times.7z
- (15.32 MiB) Downloaded 160 times
New special mpls structure
Disk is in paw. Investigating...
New special mpls structure
00006.mpls is really weird. The duration of 00005.mpls is correct for 00001.m2ts: 1:34:31. But the 00006.mpls defines the INtime and OUTtime for M2TS such that the duration of {20 x 00001.m2ts} is only 00:25:29.
OK, perhaps 00006.mpls it is a looping menu of a small piece of the main M2TS. I suppose it could be true and that could be a valid reason to want it. I suppose that the 'strict playlist' option would be applicable here. But I'm thinking that that option may not be taking into account the -collapse option. I'll look into that.
OK, perhaps 00006.mpls it is a looping menu of a small piece of the main M2TS. I suppose it could be true and that could be a valid reason to want it. I suppose that the 'strict playlist' option would be applicable here. But I'm thinking that that option may not be taking into account the -collapse option. I'll look into that.
New special mpls structure
Yup, my theory was correct. The content for that playlist is a loop of the opening.
Try this build. Set the -collapse and -sp options.
https://rationalqm.us/misc/DGDemux_hubblec4.zip
One thing I could improve is to terminate when the OUTtime is reached instead of parsing through the whole file. But it's not simple, because I don't know which ES is going to finish last, so I probably won't do anything for that. Sure, I could wait for all the streams to reach the OUTtime but it's extra complexity for little gain.
Try this build. Set the -collapse and -sp options.
https://rationalqm.us/misc/DGDemux_hubblec4.zip
One thing I could improve is to terminate when the OUTtime is reached instead of parsing through the whole file. But it's not simple, because I don't know which ES is going to finish last, so I probably won't do anything for that. Sure, I could wait for all the streams to reach the OUTtime but it's extra complexity for little gain.
New special mpls structure
Hi Rocky.
I had many work today and I will test and answer at night.
I had many work today and I will test and answer at night.
New special mpls structure
OK, good.
Why not?But for this mpls the -collapse option should not be used.
I have to think about that. I'm pretty sure many disks are like that and if it was a problem I'd have heard about it by now due to the extra content or async it would cause. I'll look through my large disk collection to see if I can find any like that, and see how DGDemux deals with it. And why aren't you seeing issues with your multi-edition blurays? Don't worry, I'll get to the bottom of it.Yes, it looks like DGDemux should only use data from the INtime to the OUTtime for each play_item.
What if the play_item list consisted of different m2ts files (just as is the case with a multi-edition Blu-ray), but each play_item does not use the entire file, but only a few seconds or minutes like the 00006.mpls?
The key question is whether there is actually any data outside the INtime/OUTtime other than the weird case of the Kino Lorber stuff at the start of the first playlist. And even if there is, is it valid useful data? Let's see. I'm going to need a problem to be demonstrated; I'm not going to do anything on hypotheticals.
If I wasn't already so busy, I could make a utility that scans an MPLS and looks for data outside the time window of each M2TS in the playlist.
New special mpls structure
The bluray spec seems to support your stance:
But it talks only about timestamps (which would be used for playback) and is silent about the presence of data outside the windows. Again, I would have surely received problem reports if there was such data. I'm going to make the utility I talked about because I am super curious now. If I have to do it it's not difficult. We'll see.The PlayItem specifies a time based playing interval from the IN_time until the OUT_time. ...
Each PlayItem shall refer to a Clip/Clips stored in the same BDMV directory as the PlayList file exists.
When a PlayList is composed of two or more PlayItems and PlayList_playback_type field in
associated AppInfoPlayList() of the PlayList is set to 1 (“Sequential playback of PlayItems”), the
playing intervals of these PlayItems shall be placed in line without a time gap or overlap on a Global
time axis of the PlayList (see Figure 5-2).
New special mpls structure
I hacked up the utility and ran it on your 00006.mpls. It detected data after the out time but no data before the in time. That agrees with the fact that it is a loop of the opening. I ran it on two random disks from my collection and it detected no data outside the windows for any M2TS in the playlist. I need to run it on some more disks, of course.
Actually the extra data may not cause async if there is extra audio and video data, but the extra content would be noticeable when playing. I have never had a report of extra content. I just don't think it would ever be present for a main movie. What would be the point of putting it there? For menu stuff, OK, I could see that, as for your case.
I'll report on further testing tomorrow. I could also give you the utility.
Actually the extra data may not cause async if there is extra audio and video data, but the extra content would be noticeable when playing. I have never had a report of extra content. I just don't think it would ever be present for a main movie. What would be the point of putting it there? For menu stuff, OK, I could see that, as for your case.
I'll report on further testing tomorrow. I could also give you the utility.
New special mpls structure
Playing the devil's advocate...
If it is true that there is never data outside the windows for a main movie demux, then it would be harmless to always honor the windows, i.e., it would effectively be a no-op. So why not just go ahead and always honor the windows? It would be the same as our strict playlist option, but applied to every M2TS in the playlist. Seems reasonable.
The fly in the ointment is this thread:
https://www.rationalqm.us/board/viewtop ... =16&t=1259
I'm going to re-visit that problem, as possibly some bug in the application of -sp was involved, rather than the concept itself. I've ordered the disk but it will take 2 weeks to get here. Meanwhile I'll test disks from my collection with -sp to see if I find any issues.
I'm tending towards wanting to honor the windows as you suggest, but not if it is going to break some disks.
If it is true that there is never data outside the windows for a main movie demux, then it would be harmless to always honor the windows, i.e., it would effectively be a no-op. So why not just go ahead and always honor the windows? It would be the same as our strict playlist option, but applied to every M2TS in the playlist. Seems reasonable.
The fly in the ointment is this thread:
https://www.rationalqm.us/board/viewtop ... =16&t=1259
I'm going to re-visit that problem, as possibly some bug in the application of -sp was involved, rather than the concept itself. I've ordered the disk but it will take 2 weeks to get here. Meanwhile I'll test disks from my collection with -sp to see if I find any issues.
I'm tending towards wanting to honor the windows as you suggest, but not if it is going to break some disks.
New special mpls structure
I spot checked some disks. All tested (with one exception) have nothing outside the window for all M2TSs. E.g., CARS_2 with 75 M2TSs had no data outside the window for all the M2TSs. But I did find one that has extra data at the end of the movie (this disk has only one M2TS). The amount of extra data was small, just some black frames at the end of the movie.
But in the linked thread, about 40GB was lost with -sp! I'll keep looking at all my disks, while waiting for that disk to arrive.
But in the linked thread, about 40GB was lost with -sp! I'll keep looking at all my disks, while waiting for that disk to arrive.
New special mpls structure
If the 00001.m2ts file is only used once, then all other data for the following PlayItems will be missing, even if it is the 00001.m2ts file again.
This wouldn't make sense for a "normal" menu mpls because the entire contents of the same m2ts file are always used, but not in this case.
MTX also takes a certain amount of data from all PlayItems and puts them together. This gives us a file with a playing time of 25 minutes. The scenes used are not the same.
Such an mpls structure is also completely new for me and I still have to think about how best to handle the whole thing.hubblec4 wrote: ↑Sun Jan 28, 2024 3:34 pmI have to think about that. I'm pretty sure many disks are like that and if it was a problem I'd have heard about it by now due to the extra content or async it would cause. I'll look through my large disk collection to see if I can find any like that, and see how DGDemux deals with it. And why aren't you seeing issues with your multi-edition blurays? Don't worry, I'll get to the bottom of it.Yes, it looks like DGDemux should only use data from the INtime to the OUTtime for each play_item.
What if the play_item list consisted of different m2ts files (just as is the case with a multi-edition Blu-ray), but each play_item does not use the entire file, but only a few seconds or minutes like the 00006.mpls?
The key question is whether there is actually any data outside the INtime/OUTtime other than the weird case of the Kino Lorber stuff at the start of the first playlist. And even if there is, is it valid useful data? Let's see. I'm going to need a problem to be demonstrated; I'm not going to do anything on hypotheticals.
At the moment cE cannot integrate these mpls into a multi-edition MKV. But it should be possible.
And yes, it is only hypothetical for now and there is no need for action at the moment.
New special mpls structure
As always, you are very hardworking.Rocky wrote: ↑Sun Jan 28, 2024 8:19 pmI hacked up the utility and ran it on your 00006.mpls. It detected data after the out time but no data before the in time. That agrees with the fact that it is a loop of the opening. I ran it on two random disks from my collection and it detected no data outside the windows for any M2TS in the playlist. I need to run it on some more disks, of course.
Actually the extra data may not cause async if there is extra audio and video data, but the extra content would be noticeable when playing. I have never had a report of extra content. I just don't think it would ever be present for a main movie. What would be the point of putting it there? For menu stuff, OK, I could see that, as for your case.
I'll report on further testing tomorrow. I could also give you the utility.
Since I have very very little knowledge about the m2ts format, any aid would be good. I just don't know if I'm interpreting and understanding everything correctly.
Until now I assumed that these IN and OUT times simply defined an area that the player should play. Just like the ordered chapters work at Matroska.
New special mpls structure
Rocky wrote: ↑Mon Jan 29, 2024 1:27 amPlaying the devil's advocate...
If it is true that there is never data outside the windows for a main movie demux, then it would be harmless to always honor the windows, i.e., it would effectively be a no-op. So why not just go ahead and always honor the windows? It would be the same as our strict playlist option, but applied to every M2TS in the playlist. Seems reasonable.
I'm going to re-visit that problem, as possibly some bug in the application of -sp was involved, rather than the concept itself. I've ordered the disk but it will take 2 weeks to get here. Meanwhile I'll test disks from my collection with -sp to see if I find any issues.
I can't say I understand it all, but it sounds like the right direction.
Yes, the same applies to cE, I'll definitely have to modify a few things, but only as long as it doesn't break anything.
New special mpls structure
He he. Out of 63 disks, 8 have data outside the windows. I've looked at two of them so far and they are innocent. I'll give a full report when I finish looking at all of them.
But! When looking at one I found a major bug in -sp that explains the big data loss in the thread I linked earlier. The end time of the movie was not being picked up properly and was random.
So I'm going to get out a fixed build ASAP with these:
* Fix major bug in -sp.
* Fix -sp not playing nice with -collapse.
Then I will come back to this thread. It's looking likely that I will honor the windows for all M2TS by default, with an option to not do so. Who knows, maybe someone wants the color bars at the start of some movies.![Razz :P](./images/smilies/icon_razz.gif)
But! When looking at one I found a major bug in -sp that explains the big data loss in the thread I linked earlier. The end time of the movie was not being picked up properly and was random.
![Embarrassed :oops:](./images/smilies/icon_redface.gif)
So I'm going to get out a fixed build ASAP with these:
* Fix major bug in -sp.
* Fix -sp not playing nice with -collapse.
Then I will come back to this thread. It's looking likely that I will honor the windows for all M2TS by default, with an option to not do so. Who knows, maybe someone wants the color bars at the start of some movies.
![Razz :P](./images/smilies/icon_razz.gif)
New special mpls structure
After extensive analysis of my disk collection (taking more than 8 minutes -- inside joke), I have concluded that we should honor IN/OUT times for all M2TSs in the playlist. This should be done by default and an option to just demux everything should also be provided. Someone might really want those color bars and Kino Lorber meta screens. ![Wink ;)](./images/smilies/icon_e_wink.gif)
In the case of your COLUMBO 00006 playlist, with this change you would get the intro repeated 20 times, unless you set -collapse in which case you'd get it once. I believe this is what you are asking for.
I've studied the consequences of doing this change for every disk in my collection and all it would do is remove color bars, Kino Lorber meta screens, and black frames. That is the stuff that players don't play, and the bluray spec explicitly says to ditch it. So Bob's yer uncle.
Interestingly, the only disk whose result would change from doing this instead of the existing -sp is your COLUMBO 00006.
I'll implement this and do careful testing with y'all's help before releasing it.
![Wink ;)](./images/smilies/icon_e_wink.gif)
In the case of your COLUMBO 00006 playlist, with this change you would get the intro repeated 20 times, unless you set -collapse in which case you'd get it once. I believe this is what you are asking for.
I've studied the consequences of doing this change for every disk in my collection and all it would do is remove color bars, Kino Lorber meta screens, and black frames. That is the stuff that players don't play, and the bluray spec explicitly says to ditch it. So Bob's yer uncle.
Interestingly, the only disk whose result would change from doing this instead of the existing -sp is your COLUMBO 00006.
I'll implement this and do careful testing with y'all's help before releasing it.
New special mpls structure
Further data on your COLUMBO 00006.mpls. It is not in fact a looping of the opening. The playlist repeats the 00001.m2ts twenty times, but each one has a different INtime/OUTtime:
1st clip is IN 4199, OUT 4412 (in seconds)
2nd clip is IN 4425, OUT 4514
3rd clip is IN 4735, OUT 4816
etc.
So this playlist is some kind of assembling of random parts of the movie. The results of demuxing with -sp without -collapse will not be a repeat of the opening 4199-4412.
I have a local build honoring windows for each M2TS and it faithfully follows the above assembling. That's how I discovered that it's not a looping of the opening. So, while that playlist appears useless, it's good for testing my algorithm.![Rolling Eyes :roll:](./images/smilies/icon_rolleyes.gif)
Demuxing it with -sp but without -collapse takes a long time (while still only writing the correct amount of 5GB). I don't want to try to break out of each M2TS early because I'd have to track all the streams to make sure they all passed the OUT time and that is too much complexity just to make this freaky scenario faster. Any extra complexity like that is going to be risky for ordinary playlists. And for this playlist, as the per-clip windows advance into the movie, early exit becomes less useful.
1st clip is IN 4199, OUT 4412 (in seconds)
2nd clip is IN 4425, OUT 4514
3rd clip is IN 4735, OUT 4816
etc.
So this playlist is some kind of assembling of random parts of the movie. The results of demuxing with -sp without -collapse will not be a repeat of the opening 4199-4412.
I have a local build honoring windows for each M2TS and it faithfully follows the above assembling. That's how I discovered that it's not a looping of the opening. So, while that playlist appears useless, it's good for testing my algorithm.
![Rolling Eyes :roll:](./images/smilies/icon_rolleyes.gif)
Demuxing it with -sp but without -collapse takes a long time (while still only writing the correct amount of 5GB). I don't want to try to break out of each M2TS early because I'd have to track all the streams to make sure they all passed the OUT time and that is too much complexity just to make this freaky scenario faster. Any extra complexity like that is going to be risky for ordinary playlists. And for this playlist, as the per-clip windows advance into the movie, early exit becomes less useful.
New special mpls structure
Yes, exactly, it's not a menu playlist in the actual sense as it was intended, and that's why the -collapse switch is there to prevent the exact same content from being strung together multiple times.Rocky wrote: ↑Tue Jan 30, 2024 9:51 amFurther data on your COLUMBO 00006.mpls. It is not in fact a looping of the opening. The playlist repeats the 00001.m2ts twenty times, but each one has a different INtime/OUTtime:
1st clip is IN 4199, OUT 4412 (in seconds)
2nd clip is IN 4425, OUT 4514
3rd clip is IN 4735, OUT 4816
etc.
So this playlist is some kind of assembling of random parts of the movie. The results of demuxing with -sp without -collapse will not be a repeat of the opening 4199-4412.
But here different content is always used for each PlayItem.
I also had the thought that this mpls is rather "useless", or rather serves as a "confusion" to confuse tools like DGDemux and cE. I've already had a few discs where such methods were used.
A Blu-ray with over 300 different mpls files, all of which have the same playback length and play the main film (which was apparently split into multiple m2ts files specifically for this purpose) in different ways. the m2ts files are randomly distributed in each mpls.
The whole thing was discussed in the makemkv forum.
On the other hand, I find the way the film is only played in bits and pieces interesting and could be viewed as a "preview of the best scenes".
Sounds like you nailed it.Rocky wrote: ↑Tue Jan 30, 2024 9:51 amDemuxing it with -sp but without -collapse takes a long time (while still only writing the correct amount of 5GB). I don't want to try to break out of each M2TS early because I'd have to track all the streams to make sure they all passed the OUT time and that is too much complexity just to make this freaky scenario faster. Any extra complexity like that is going to be risky for ordinary playlists. And for this playlist, as the per-clip windows advance into the movie, early exit becomes less useful.
For me it's OK if it's not super fast and DGDemux waits until the end of the file.
I also see it like this, it's more important that nothing else gets broken.
Even if MTX handles these mpls correctly, I'm still thinking about how I can incorporate this into cE.
In principle, it really doesn't make any sense to convert these mpls to an MKV because then the episode simply doesn't fit anymore and wouldn't make any sense.
BUT as a second edition in a multi-edition MKV would be the best in this case.
Either way, the user will convert the entire 00001.m2ts into an MKV, and thus the content for the mpls 00006 is also available. All cE would then have to do is create a corresponding chapter.xml.
This means that this mpls would not have to be demuxed.
But I had already checked whether my current heuristic could do that, but unfortunately not.
New special mpls structure
I agree with you that it's probably obfuscation. I wouldn't bother doing anything with it. eac3to doesn't even show it unless you give -showall.
Just to be clear, I'm still going ahead with honoring the per-clip windows, because that's what the bluray spec calls for. I will default to what used to be -sp, and which will now be -ignorepl for ignore playlist in/out times.
Just to be clear, I'm still going ahead with honoring the per-clip windows, because that's what the bluray spec calls for. I will default to what used to be -sp, and which will now be -ignorepl for ignore playlist in/out times.
- SomeHumanPerson
- Posts: 96
- Joined: Fri Mar 24, 2023 10:41 am
New special mpls structure
Not sure if I followed every single detail in this thread, but if I'm at least understanding the COLUMBO issue correctly, that playlist is a cleverly-done "montage" and now that we've seen it, I'm more than a little shocked that this is the first time anyone has encountered one. That seems like an excellent way to do menu or "idle" video segments without anyone at the studio having to actually edit/produce a separate video.
I do think it's very cool that this whole situation coincidentally managed to uncover an existing but mostly unrelated bug so that it could be squashed!
I do think it's very cool that this whole situation coincidentally managed to uncover an existing but mostly unrelated bug so that it could be squashed!
New special mpls structure
I tested eac3to.
The duration of the mpls is 0:25:29 in the first log, but after selecting the first title "1)" the duration is now over 31h.
And the demuxed streams also very big.
----
Unfortunately I no longer have a link to the thread and unfortunately I also forgot the name of the film. It was from the times when madshi supported eac3to.
But maybe this link will help.
https://forum.makemkv.com/forum/viewtop ... =8&t=14330
Code: Select all
"D:\eac3to.exe" "G:\COLUMBO_DISC24\BDMV\PLAYLIST\00006.mpls"
1) 00006.mpls, 0:25:29
[1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1].m2ts
- Chapters, 23 chapters
- h264/AVC, 1080i60/1.001 (16:9)
- AC3, English, stereo, 48kHz
- AC3, Russian, stereo, 48kHz
- AC3, English, stereo, 48kHz
Code: Select all
"D:\eac3to.exe" "G:\COLUMBO_DISC24\BDMV\PLAYLIST\00006.mpls" 1)
M2TS, 1 video track, 3 audio tracks, 3 subtitle tracks, 31:30:39, 60i /1.001
1: Chapters, 23 chapters
2: h264/AVC, 1080i60 /1.001 (16:9)
3: AC3, English, 2.0 channels, 448kbps, 48kHz, dialnorm: -27dB
4: AC3, Russian, 2.0 channels, 192kbps, 48kHz
5: AC3, English, 2.0 channels, 448kbps, 48kHz, dialnorm: -27dB
6: Subtitle (PGS), Japanese
7: Subtitle (PGS), Japanese
8: Subtitle (PGS), English
And the demuxed streams also very big.
----
Unfortunately I no longer have a link to the thread and unfortunately I also forgot the name of the film. It was from the times when madshi supported eac3to.
But maybe this link will help.
https://forum.makemkv.com/forum/viewtop ... =8&t=14330
New special mpls structure
Sounds great Rocky, and I hope it is not so much work for you.
New special mpls structure
It's almost done already.
I'm not surprised eac3to gets it wrong. We haven't looked at that yet.
I'm not surprised eac3to gets it wrong. We haven't looked at that yet.
New special mpls structure
Sorry I misunderstood this. We talk via email because he is still waiting to be activated by Doom9.
New special mpls structure
I see, thank you.
New special mpls structure
Everything works except that it broke subtitles. I'll try to fix that tomorrow.