It took a little bit of effort but the stange characters that WordPress was displaying through a significant number of posts have now been removed.
I went looking briefly about what might have caused the problem and can confirm that it is related to the character encoding in MySQL. There are numerous posts in the official WordPress support forums and on blogs dating back to 2006 – so it isn’t a new problem at all.
After removing them from the site, I’ve subsequently verified that the database encoding is UTF-8 and WordPress is publishing in UTF-8 according to the content type in the meta data. That makes me think that those strange characters could have been present in my site from an earlier upgrade of WordPress and I’ve just never noticed it before.
I think this might be karma for not doing enough housekeeping on my site.