Codec shoot-out planning

[Audionut]

I've got 5 weeks holiday starting next month.  2 weeks planned, 3 weeks looking for things to do.

I've got a few 1080p sources with varying amounts of grain, some clean ones like kung-fu panda.

I'll need download links for encoders other than x264, xvid etc.

Suggestions for content, settings etc.

Anything else you can think of.

[nakTT]

Its been a while since the last shoot-out. Looking forward to your report/result.

[Dark Shikari]

1) Start by checking all the encoders I listed during my last test.

2) Post a thread detailing your planned methodology before you do it, including encoder settings and so forth, before you do it.  This will let people critique it and point out problems.

3) I'd personally suggest focusing on sources that aren't too grainy and have a good amount of detail.  Testing grain retention is often pretty boring because it flattens the difference between encoders, making the results of the comparison less meaningful.  The HD test clips here are good examples of ultra-high-detail test clips with minimal grain.

[Mixer]

Any chance you could test x86 vs x64 versions in your shootout?

[Audionut]

1) I've gathered, x264, xvid, real, badaboom, wmv, ffmpeg, nero, quicktime, vp7, elecard.

2) Well x264 is easy with presets.  I'll probably stick to veryfast, medium, veryslow.  With tune psnr and tune grain.  It would be nice if you could provide a visual quality setting.
I'll need suggestions on settings for other codecs!!  I've only just installed them.

3) I'm thinking, park_joy, rush_hour & something from star trek (blu-ray) - all resized to 720.

[Audionut]

Any chance you could test x86 vs x64 versions in your shootout?

Doubtful as the only difference would be speed.  Although I could probably take one setting and encode with x64 to show speed difference.  Think very low priority.

[Dark Shikari]

Doubtful as the only difference would be speed.  Although I could probably take one setting and encode with x64 to show speed difference.  Think very low priority.

You should make sure you have a mission for each test: i.e. what it intends to actually test.  For example, you could go with "the maximum possible quality, ignoring speed", and use insane settings for all codecs.

[Audionut]

Best quality.  So I guess I should let x264 use placebo.

I think I should really use psnr and ssim for others who would get linked to the review and not trust their eyes.  But I'm not so sure now.
I'm leaning towards best visual quality settings for each encoder only,
a) reduce time producing the shoot-out,
b) might be able to convince people metrics mean shit.

[Dark Shikari]

Best quality.  So I guess I should let x264 use placebo.

I think I should really use psnr and ssim for others who would get linked to the review and not trust their eyes.  But I'm not so sure now.
I'm leaning towards best visual quality settings for each encoder only,
a) reduce time producing the shoot-out,
b) might be able to convince people metrics mean shit.

x264 probably the only video encoder you'll test with such heavy psy optimizations, so I doubt it'll convince people that metrics are meaningless, since x264 is also the best at PSNR and so forth.  Here's what I'd do:

1.  Do a PSNR or SSIM test, with all codecs tuned towards that when possible.
2.  Do a visual test, with screenshots, with all codecs tuned towards visual quality.

Doing PSNR/SSIM testing with psy optimizations is at best misleading and at worst going to invite completely wrong conclusions from people simply looking at graphs and nothing else.

The first will let you get pretty graphs, while the second will let you get pretty screenshots.

Another thing to note is you may want to do multiple bitrates, especially for the visual test.  The reason here is that it's easy to look at two screenshots and say: "oh, x264 is barely better than Xvid", despite the fact that Xvid might have needed twice the bitrate to look as good as x264.  It's more insightful to be able to see what bitrate with X you need to roughly equal Y.

[Shinigami-Sama]

if you get brave you could do some double-blind tests with short lossless clips/PNGs on the visual tests

[Audionut]

Sounds good.  Perhaps with these sources, 1000, 2000 and 4000 kbps.

Also thinking of using avisynth compare with ffms2 as the source filter to measure psnr.

[CryptWizard]

double-blind

Yes! This.
But even single blind should be sufficient given the nature of the internet (can't show emotions).
And yeh, encode the clips back to x264, HuffYUV or Lagarith lossless before showing the judge.

[Audionut]

Blind test is a good idea.  Adds another result to the test.

I think the only decent encoder i'm missing is ateme.

If there is someone from Ateme who would be willing to provide a binary perhaps time limited please pm me.  I can't find it with my regular sources to use for this test.

[Dark Shikari]

Blind test is a good idea.  Adds another result to the test.

I think the only decent encoder i'm missing is ateme.

If there is someone from Ateme who would be willing to provide a binary perhaps time limited please pm me.  I can't find it with my regular sources to use for this test.

Even I don't have a recent enough Ateme to use in a fair test.  They didn't give me a recent one either.

Now, you might be able to convince one to do the encoding for you and give you samples.

[D404]

Am I the only one that remembers Xvid AVC? Whatever the heck happened to that (it was in the last doom9 shoot-out).

[saintdev]

I was thinking about putting together a public visual quality test similar to what hydrogen audio does for their audio tests. The major hindrance to this is the lack of ABX software for video.

[DarkZell666]

Am I the only one that remembers Xvid AVC? Whatever the heck happened to that (it was in the last doom9 shoot-out).

Now that's a good question ... o_O

[RiCON]

I remember someone thinking of making a Video ABX program. Can't remember if here on Doom10 or on Doom9.

[cogman]

Am I the only one that remembers Xvid AVC? Whatever the heck happened to that (it was in the last doom9 shoot-out).

x264 happened, and the Xvid project pretty much died all together.

I would like to see a speed graph if available. I know it isn't everything, but I would like to see how x264's veryfast compares to other encoders speeds.

I know this isn't an x264 review, but it would be nice to see how x264's tune settings affect different videos (or the same video), just to get an idea of what is going on when you say --tune grain

[D404]

x264 happened, and the Xvid project pretty much died all together.

x264 was already around (and won said codec competition also iirc), and Xvid was already dead. Yet, they announced Xvid AVC. :V

[Forteen88]

1) I've gathered, x264, xvid, real, badaboom, wmv, ffmpeg, nero, quicktime, vp7, elecard.

Get DivX (Pro?) encoder also, although I don't know if DivX-encoder is only a h264-encoder nowadays?

[Audionut]

Get DivX (Pro?) encoder also, although I don't know if DivX-encoder is only a h264-encoder nowadays?

Yeah, Just had that thought when I seen divx 6.8 in my vfw list.   

[Audionut]

hmm, how to adjust the setting of the divx7 converter?

edit:  nevermind, I RTFM.

[Audionut]

Sources will be,

park_joy (16:9 | 500 frames |
rush_hour (16:9 | 500 frames |

from here:  http://media.xiph.org/video/derf/

And these 2,
http://members.iinet.net.au/~audionut/star1.mkv
http://members.iinet.net.au/~audionut/star4.mkv

edit: the star trek sources are re-encoded to minimize upload time.
Original source from Blu-ray.

[Dark Shikari]

Rush Hour, really?  I've always hated that source because it's so blurry and out of focus that it's nearly impossible to do visual comparisons on.

[shon3i]

Hello all, this topic is a big reason why I registered on Doom10  , i love codec shoot outs and but i have mine opinions which i want to present.

First i want to say the winner of this test .... x264 second i want to see other H264 implementations such Mainconcept, Ateme especialy. Audionut please contact me via pm i can give you access tougt recent versions both anteme and mainconcept, but with mainconcept there is one restriction, because i have it only via scenarist which is restricted to Blu-Ray profile. So i want to suggest if we maybe can follow only Blu-Ray restriciotns for all codecs and to actually see what they can. For example to put 1080p video to BD5-9 where averge bitrate should be around 4000-8000kbps. I think is fair enough for all implementation.

EDIT: Since this is multicodec and multiformat blu-ray is not good idea, and we need only best from one group. Time vs Quality test should be nice too.

[poisondeathray]

Not trying to make life any harder , but are those test clips (500 frames) long enough?

Will you even out the keyframe interval between the encoders? e.g. the default interval for most mainconcept h.264 based encoders is 33, but x264 has 250.

I'm sure there are other little variables that may skew the comparison if not controlled for

[Dark Shikari]

Not trying to make life any harder , but are those test clips (500 frames) long enough?

Will you even out the keyframe interval between the encoders? e.g. the default interval for most mainconcept h.264 based encoders is 33, but x264 has 250.

I'm sure there are other little variables that may skew the comparison if not controlled for

It's easy to equalize the keyframe interval between encoders; obviously that has to be done.

I do agree that the "standard test clips" are generally overly short; the one good thing about them is that because there's no scene changes, you can pick any frame and be confident it's a relatively fair comparison.  On the other hand, with a much longer test clip with lots of scenes, the quality may vary significantly between scenes.  Of course, one does still need a longer test clip or two.

[poisondeathray]

I have a sort of related question that I hope someone can answer:

What is the "proper" procedure in terms of using colormatrix() and rawsource() in avisynth for those http://media.xiph.org/video/derf/
 test sequences?

The readme for most of the HD sequences say Rec.709 was used for the workflow

Is RawSource() alone enough for those y4m and yuv sequences?

or would you do something like: ?

RawSource("park_joy_1080p.y4m")
ColorMatrix(mode="rec.601->rec.709", clamp=0)

The documentation for colormatrix isn't clear on this, and usually only refers to DVD. Thanks for any insight

PS. If this isn't a good place to ask, I can post at the other board in the colormatrix thread...

[Audionut]

Ok, ditching rush_hour for Big Buck Bunny.

Also, I'm going to keep the Star Trek sources, as despite the "problems" using an all ready encoded source, IMO, it represents a real case scenario, where a user rips a blu-ray and shrinks it.