This is an unusual setup, and it sounds like the issue is more to do with the presence/absence of a network connection than anything else. Though from the way you've stated your conditions, you say that if you compare the following:
a) profiling with Scout, without advanced telemetry enabled on the SWF
b) profiling with Scout, with advanced telemetry enabled on the SWF
then (b) has a much higher framerate than (a)? That seems hard to believe - if you're profiling with the same options (e.g. just basic telemetry) then the only difference the advanced telemetry setting makes, is for Flash Player to parse an extra tag on the SWF. There should be no functional difference.
If I were you, I'd do some more tests without Scout, such as running some other content that uses TCP/IP (pretty much anything that has a Loader in it) at the same time as your video content - that would narrow it down to whether there's some strange condition where Flash Player performs badly if it's not using a network socket. Even better, try opening a socket within your content (flash.net.Socket is a TCP socket), and sending/receiving some data - as far as the network is concerned, that's all Flash Player is doing when it's connected to Scout.
I won't rule out the possibility a bug of this nature, since it's an unusual configuration, and not as tested as more common configurations. If you can narrow it down like this, that would help; in either case, I'd file a bug:
The only other thing I'd say, is that I wouldn't lose any sleep over the experience of your users - I doubt anyone will have the same setup, and even if they did, they could switch back to OS X.