I still can't see how the regex can now which font closing tag it should parse?
Maybe my example was a bad one, since you could merge them into one markup tag like you have. But that wasn't the point. The point being that if different markups both have the </font> the regex will have no way of knowing which one is the one that it should match to. Some amazing things can be done with regexes, but magic isn't one of them. =]
Testing your code shows that the following happens:
Initially:
<font style="courier"><font color="red"><font size="5"> Red courier, size 5 </font></font></font>
After iteration 1:
[font style="courier"]<font color="red"><font size="5"> Red courier, size 5 [/font]</font></font>
After iteration 2:
[font style="courier"][font color="red"]<font size="5"> Red courier, size 5 [/font][/font]</font>
After iteration 3:
[font style="courier"][font color="red"][font size="5"] Red courier, size 5 [/font][/font][/font]
In this case it doesn't matter that much of course, but it shows that the replace is done with the first possible match. If the </font> tag is part of something else, it will break, and it could potentially screw up the nesting as well.