Solved
Took Ned's advice below and used a thin layer of grease (plumber's grease so as not to attack the nitrile) to lubricate the thermostat housing walls. The BS134 O-ring fit onto the cap and I seated it to the base of the caged flange (see picture).

I replaced the cap and gave it a slight, 90 degree twist as I pushed it home, aligning the screw holes in one go. The unit felt "seated" with this operation, unlike before. I think the previous try might have warped or kinked the o-ring. The grease made the o-ring slide and not bind. I secured the cap screws and tested. No leaks. Thanks for the tip about the grease. As interest, the o-ring seats against a lip on the inside of the thermostat housing (see picture).

The dimension of the ring ensures exactly the right amount of pressure against the thermostat gasket and exactly the right deformation of the o-ring to seal the assembly.

Original problem
As part of diagnosing my earlier cooling problems, I cracked open the thermostat housing on my K100RT to replace the thermostat. I substituted the original for a new one but discovered that there was nothing wrong with the old thermostat. I started to have coolant leaks from the thermostat cover. So, I reopened the housing, replaced the new thermostat for the original (thought it might be due to the non-BMW thermostat I used) and replaced the O-ring on the cover. I used an aftermarket BS134 o-ring, not an original BMW part. I am still getting leaking from the cover. Refering to the image below:


The o-ring I fitted seats between the thermostat housing and the cap (3) on the inside edge, indicated with the RED arrow. Should it be seated on the outside edge? What can I do to stop this problem? My novice mechanical abilities certainly show up here. This should be a no brainer.

Robert, I am not sure about which BS O ring it is, but yes you are correct; it is not a metric size.

I did this a few months ago and in the end I fitted the O ring over the cap(3) flange first after smearing some grease on it first. It should go past the slotted flange.

The trick was to assemble the whole thing and slowly and equally tighten the bolts ( a few turns at the time) until the o ring is squeezed into the radiator hole. Remember the thing has to only hold 1 atmosphere pressure, so don't tighten the thing too much.

O'rings are made of different material depending of their application. (static, dynamic, hot application, water or petroleum liquid....).

The original K100 prototype had a lot of problems with the seals and BMW had to have some made specially for this engine. (the ones in the water/oil pump are specials. They are not made from the regular standard rubber.)

Is the type of O'ring that you bought made to be used with a temperature in excess of 100 degrees?
The OEM O'ring should not be too expensive and I would advise to buy one.

Crazy Frog wrote:Is the type of O'ring that you bought made to be used with a temperature in excess of 100 degrees?

The ring I bought is a nitrile rubber ring which conforms to ASTM D2000, M2BG714, B14, EA14, EF11, EF21, EO14, EO34. I don't have these specs but one of them will tell me the temperature rating of this product. I presume that the nitrile material and the Durometer A rating would give it the required heat tolerance and elasticity to conform to the surfaces which it contacts.

Crazy Frog wrote:The OEM O'ring should not be too expensive and I would advise to buy one.

$0.50 versus $10.00 each ring is motivation for me to try the no-name brand over BMW's brand.

My problem I guess is that I cannot get the ring to seal reliably and properly. It doesn't appear to be a failure mode due to wrong material. As soon as the water reaches 89 C and the thermostat opens, the leak appears. When the water cools and the thermostat closes after shutting off the bike, the leak dries up.

The ring is positioned, at the moment, at the base of the slotted flange that applied pressure to the thermostat. I have to refit the ring to drop into the recess in the cap, as suggested by Ned. This will force the ring between the cap and the outside edge of the housing, forming a water tight seal, I guess. I'll try the careful reseating and plumber's grease as recommended by Ned and, if that fails to halt the leak, I have the BMW ring on order. I only lost time and $0.50.