Exhaust manifold flange question

Hi, quick question when replacing my exhaust pipes. Is RTV used when mating the flanges together? I took a wire wheel to the exhaust flanges, which look like the image below, and cannot get all of the shit off. When reconnecting, should I strive for as smooth a surface as possible; or is this not as important since we use the metal gasket inline?

Edits:
1. I've read that RTV upstream of the O2 sensor can cause issues, which is why I'm doubting this could be a sealant. Is it just a layer of metal from the old cat pipe flange?
2. It might be rust-colored because I sprayed penetrant on it once I disconnected the pipes. I was so shocked when both sides came apart once the bolts were out that I forgot to take a picture.

 

I wouldn't use rtv or any sealant on it. I never used any kind of sealant on exhaust. I'd aim for a smooth surface. Could be hardened carbon deposits or some other by product. Get an abrasive softer than the metal itself and work whatever it is off there.

 

@dirtydeeds