Ok - I have had another play.
I recompiled libx264 and ffmpeg without the Atom detection, and the frame rate immediately bumped to 7 fps - still not quite 10.
You were absolutely right - the CPU was pegging - but just for a moment. I had to run it a few times to see it (due to the resolution of TOP/Activity Monitor). I think it was most certainly overloaded. It still died after about 10 seconds with some form of '[libx264] error GOP corruption'.
So, I reverted back to the default libx264/ffmpeg build and actually went to my camera and tuned the resolution down to 640x480@5fps. I did this to reduce the CPU load, because I thought perhaps the CPU load was causing, as you said, buffers to over/under flow.
The encode now works perfectly, it consumes about 25% on each of the 4 Hyper-threads. This is pretty much what you said it should. I wouldnt want to use much more power than this, because it is running 24x7.
The irony now is, if I have to tune the source down to 640x480 to make an acceptable CPU load, then I may as well just use -vcodec copy because I'm not getting my 1:1 iPhone 4 pixel ratio anyway!! In which case it uses 5% CPU because ffmpeg isnt really doing much.
I did test VLC again (against the 1600x1200 source), and it was using about 10%-15% CPU on each core to DECODE and render the x264 stream. So its expected that it takes 10 times the grunt to down-scale and re-encode the stream to 960x640?
So, until VLC 1.2 with its httplive access filter, I'll probably stick with this. I tried compiling VLC 1.2 last night but failed miserably - the forums seem to suggest the trunk is not in a buildable state - shame. If anyone out there has a running OSX build of 1.2 (no video output needed, just command-line transcoding) that they can donate to me, please, let me know!
Final result: the ffmpeg+segmenter has been running for 12 hours now, no hiccups, 25% CPU load on Atom 330 4 Hyper threads.
Thanks again nm, and J_Darnley - hopefully this thread contains enough detail to be of use to anyone else trying to do the same thing with an RTSP stream.