OK...here's my take on the expression (but I may be wrong....lol).
I grew up in the UK and at one time it was very common to see 'brass monkeys' sitting either on the mantelpieces above a fireplace or in the hearth depending on the size. I have seen them range in size from solid brass ones at about 1" tall to hollow brass ones about 12" tall and they are always cast as a single piece.
In every example I saw the monkeys are sitting on their haunches, knees up, and are either sitting in a straight line or in a slight curve. Each of the monkeys has his hands in a classic pose and they are named according to that pose. One will have his hands over his eyes and is known as 'See No Evil', the second will have his hands over his ears and is known as 'Hear No Evil' while the third has his hands over his mouth and is known as 'Speak No Evil'. It is my belief that they date originally from the pre-Victorian era (possibly of Japanese origin)and represent a lesson in morality i.e. a person of decent morals will neither see no evil, hear no evil nor speak no evil.
Now, it is very common in the UK for a well known phrase, expression or myth to become plagiarized and develop a whole new meaning. It is quite possible that this has happened in this case. If you consider the sitting positions of the monkeys their testicles would have been in contact with, or very close to, the ground (if they were sitting outside on the ground of course). In very cold weather that would have left them frozen (possibly to the ground)and if the monkeys were to stand up they could have left their testicles behind. Hence the phrase 'Cold enough to freeze the balls off a brass monkey'.
Some examples of the monkeys can be found in the links below (some of which also point to a Japanese origin).