I don't know how that is happening, unless some uninited variables are taking on very lucky values for you.
Anyway, I demuxed the 264 out of the MKV and ran a parser on it. I get the dump below. You can see that the first IDR slice has a frame_num of 1 (must be 0 by spec). This is ILLEGAL and so your stream is invalid. I am willing to try to support these lossless files, but only if I am supplied with valid files. So I will do no more on this until I receive a valid, legal test file, preferably raw 264 rather than MKV.
-----
h264parse - mpeg4ip version 1.5.0.1
Nal length 26 start code 4 bytes
ref 3 type 7 Sequence parameter set
profile: 244
constaint_set0_flag: 0
constaint_set1_flag: 0
constaint_set2_flag: 0
constaint_set3_flag: 0
level_idc: 31
seq parameter set id: 0
log2_max_frame_num_minus4: 1
pic_order_cnt_type: 0
log2_max_pic_order_cnt_lsb_minus4: 0
num_ref_frames: 0
gaps_in_frame_num_value_allowed_flag: 0
pic_width_in_mbs_minus1: 0 (16)
pic_height_in_map_minus1: 2
frame_mbs_only_flag: 0
derived height: 96
mb_adaptive_frame_field_flag: 1
direct_8x8_inference_flag: 0
frame_cropping_flag: 0
vui_parameters_present_flag: 0
Nal length 9 start code 4 bytes
ref 3 type 8 Picture parameter set
pic_parameter_set_id: 0
seq_parameter_set_id: 0
entropy_coding_mode_flag: 0
pic_order_present_flag: 0
num_slice_groups_minus1: 0
num_ref_idx_l0_active_minus1: 0
num_ref_idx_l1_active_minus1: 0
weighted_pred_flag: 0
weighted_bipred_idc: 0
pic_init_qp_minus26: -26
pic_init_qs_minus26: 0
chroma_qp_index_offset: 0
deblocking_filter_control_present_flag: 1
constrained_intra_pred_flag: 0
redundant_pic_cnt_present_flag: 0
Nal length 486 start code 4 bytes
ref 0 type 6 SEI
payload_type: 5 user_data_unregistered
payload_size: 477
0xdc 0x45 0xe9 0xbd 0xe6 0xd9 0x48 0xb7
0x96 0x2c 0xd8 0x20 0xd9 0x23 0xee 0xef
0x78 0x32 0x36 0x34 0x20 0x2d 0x20 0x63
0x6f 0x72 0x65 0x20 0x31 0x31 0x32 0x20
0x72 0x31 0x38 0x33 0x34 0x20 0x61 0x35
0x31 0x38 0x31 0x36 0x61 0x20 0x2d 0x20
0x48 0x2e 0x32 0x36 0x34 0x2f 0x4d 0x50
0x45 0x47 0x2d 0x34 0x20 0x41 0x56 0x43
0x20 0x63 0x6f 0x64 0x65 0x63 0x20 0x2d
0x20 0x43 0x6f 0x70 0x79 0x6c 0x65 0x66
0x74 0x20 0x32 0x30 0x30 0x33 0x2d 0x32
0x30 0x31 0x30 0x20 0x2d 0x20 0x68 0x74
0x74 0x70 0x3a 0x2f 0x2f 0x77 0x77 0x77
0x2e 0x76 0x69 0x64 0x65 0x6f 0x6c 0x61
0x6e 0x2e 0x6f 0x72 0x67 0x2f 0x78 0x32
0x36 0x34 0x2e 0x68 0x74 0x6d 0x6c 0x20
0x2d 0x20 0x6f 0x70 0x74 0x69 0x6f 0x6e
0x73 0x3a 0x20 0x63 0x61 0x62 0x61 0x63
0x3d 0x30 0x20 0x72 0x65 0x66 0x3d 0x31
0x20 0x64 0x65 0x62 0x6c 0x6f 0x63 0x6b
0x3d 0x30 0x3a 0x30 0x3a 0x30 0x20 0x61
0x6e 0x61 0x6c 0x79 0x73 0x65 0x3d 0x30
0x3a 0x30 0x20 0x6d 0x65 0x3d 0x64 0x69
0x61 0x20 0x73 0x75 0x62 0x6d 0x65 0x3d
0x30 0x20 0x70 0x73 0x79 0x3d 0x30 0x20
0x6d 0x69 0x78 0x65 0x64 0x5f 0x72 0x65
0x66 0x3d 0x30 0x20 0x6d 0x65 0x5f 0x72
0x61 0x6e 0x67 0x65 0x3d 0x31 0x36 0x20
0x63 0x68 0x72 0x6f 0x6d 0x61 0x5f 0x6d
0x65 0x3d 0x31 0x20 0x74 0x72 0x65 0x6c
0x6c 0x69 0x73 0x3d 0x30 0x20 0x38 0x78
0x38 0x64 0x63 0x74 0x3d 0x30 0x20 0x63
0x71 0x6d 0x3d 0x30 0x20 0x64 0x65 0x61
0x64 0x7a 0x6f 0x6e 0x65 0x3d 0x32 0x31
0x2c 0x31 0x31 0x20 0x66 0x61 0x73 0x74
0x5f 0x70 0x73 0x6b 0x69 0x70 0x3d 0x30
0x20 0x63 0x68 0x72 0x6f 0x6d 0x61 0x5f
0x71 0x70 0x5f 0x6f 0x66 0x66 0x73 0x65
0x74 0x3d 0x30 0x20 0x74 0x68 0x72 0x65
0x61 0x64 0x73 0x3d 0x36 0x20 0x73 0x6c
0x69 0x63 0x65 0x64 0x5f 0x74 0x68 0x72
0x65 0x61 0x64 0x73 0x3d 0x30 0x20 0x6e
0x72 0x3d 0x30 0x20 0x64 0x65 0x63 0x69
0x6d 0x61 0x74 0x65 0x3d 0x31 0x20 0x69
0x6e 0x74 0x65 0x72 0x6c 0x61 0x63 0x65
0x64 0x3d 0x30 0x20 0x63 0x6f 0x6e 0x73
0x74 0x72 0x61 0x69 0x6e 0x65 0x64 0x5f
0x69 0x6e 0x74 0x72 0x61 0x3d 0x30 0x20
0x62 0x66 0x72 0x61 0x6d 0x65 0x73 0x3d
0x30 0x20 0x77 0x65 0x69 0x67 0x68 0x74
0x70 0x3d 0x30 0x20 0x6b 0x65 0x79 0x69
0x6e 0x74 0x3d 0x32 0x35 0x30 0x20 0x6b
0x65 0x79 0x69 0x6e 0x74 0x5f 0x6d 0x69
0x6e 0x3d 0x32 0x35 0x20 0x73 0x63 0x65
0x6e 0x65 0x63 0x75 0x74 0x3d 0x30 0x20
0x69 0x6e 0x74 0x72 0x61 0x5f 0x72 0x65
0x66 0x72 0x65 0x73 0x68 0x3d 0x30 0x20
0x72 0x63 0x3d 0x63 0x71 0x70 0x20 0x6d
0x62 0x74 0x72 0x65 0x65 0x3d 0x30 0x20
0x71 0x70 0x3d 0x30 0x0
string is ".E....H..,. .#..x264 - core 112 r1834 a51816a - H.264/MPEG-4 AVC codec - Copyleft 2003-2010 -
http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=0 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=0 chroma_qp_offset=0 threads=6 sliced_threads=0 nr=0 decimate=1 interlaced=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=cqp mbtree=0 qp=0."
Nal length 6 start code 4 bytes
ref 0 type 9 Access unit delimeter
primary_pic_type: 0
Nal length 3442 start code 4 bytes
ref 3 type 5 Coded slice of an IDR picture
first_mb_in_slice: 0
slice_type: 7 (I)
pic_parameter_set_id: 0
frame_num: 1 (5 bits)
field_pic_flag: 0
idr_pic_id: 1
pic_order_cnt_lsb: 8
[etc.]
-----