One of the things that can happen as some men get older is that their testosterone drops considerably. If that man is already producing more than male average amounts of estrogen from the adrenals, or if there is mild androgen insensitivity, then some estrogenic effects can take place, with smoother skin and mild breast development among them. You may also find you need viagra or something similar for a statisfying sex life -- or as time goes on, any sex life at all. After all, that is one of the reasons those little pills are so popular and are advertised so heavily. (Just remember, if you use them and have an erection lasting longer than 4 hours, be sure to call the doctor! Sorry, the idea of a man with a 4 hour erection being able to call a doctor seems highly amusing to me.
I would think that there would be so little blood left for the brain that he would be in a faint.......)
So, if you are worried and want to check it out, one thing you need to have an endocrinologist check is your relative hormonal levels, E vs. T. If they are out of whack, reatively speaking, that could explain your breast growth, but should be accompanied by a reduction in sex drive too.
Breast tissue does not grow in all men that arrive at what some people have called male menopause in the 40's and 50's, but seems to in some. Male menopause often shows up as an emotional change rather than a physical, but that has been suggested as being due to hormonal shifts in the brain. Some people claim it is not physical at all, but my endocrinologist disagrees (her husband went through it and she was as persisent as a vampire in drawing blood to chart the changes.) As we age, the brain signals different and often lesser amounts of old hormones and triggers small bits of new hormones that tells the body it is aging, slow down and start getting ready to be grandfather. It can start as early as the early 30's for men. If you think back, by the 30's you kinda settled down or matured out of the risk taking of the late teens and 20's of late puberty. That is a hormone shift. At least most guys do! People usually just say the guy is maturing, but that is often the reason for the behavioral shift. Those little bits of chemical are powerful beyond all proportion when they act on the brain!!!
Body mass index can have some effect on man boobs, and if you had well developed pecs in times past, then muscle conversion to fat cells could happen too. Excercise to make fat to muscle can help there. If a formerly well excercised body has been allowed to "go to fat" then you may get man boobs too, or being overweight and losing it fast.
Other groups that may grow man boobs in later life includes people who smoked a good bit of marijuana in their youth, and those who are mildly intersexed.
A version of Kleinfelters syndrome, an intersexed condition that may affect one in 500 males to one degree or another, could be responsible. It's a trisomy birth defect where the genetic karyotype (as revealed by blood tests) shows 47xxy sex chromosomes or some variant due to uneven splits during egg, sperm or later foetal cell division (most usually within the egg or fertilized egg, as when it occurs in the sperm the head is too large to gain access to the egg interior). The actual birth defect may be in the enzyme that unzips the two strands of chomosomes before the cell division. In any event, this birth defect produces an outwardly normal male 46xy pattern or normal female 46xx pattern, but at the genetic level the foetus is 47xxy or more.
Sometimes the defect continues to affect cell division within the foetus and a mosaic pattern where some cells may bexy, xxy and some xxxyy or xxxyyyxy or other variant. Most people who have a 47xxy and variant condition never know nor are affected but a very, very few may find themselves with late onset secondary partial female puberty (usually happens in the 40's but one case in England happened to a 63 year old male). I do know of three cases (2 Aussie, 1 California) who I have talked with, and they sent me to two others in Aussie-land. I grew up with one friend who had a mild episode with it in his late 30's and had a mastectomy, and a colleague at my work who also had a mastectomy in his '50's as the developing breasts were embarrassing to him. He also was sterile. Both are 47xxy or variant, as am I.
You may have seen on the tube or read that in Mexico, there are a few villages where secondary sex characteristics shifts seems to be common, but that may be due to Congenital Adrenal Hyperplasia or the 5 alpha reductase birth defects since the onset is often at puberty.
Having Kleinfelters is not life threatening (I am 66 and counting) though a lower than average sex drive and smaller male phalus and testicles and lower than average fertility or sterility seems common. There may also have been an early correction of a hypospadia in ones medical history, but if they are mild, they are usually corrected immediately after birth and may not even be recorded. Any breasts that may form are usually not considered a danger, though the tissue could (low probability) become cancerous, and as long as the individual isn't bothered by it, it is usually left alone. Short of masectomy there really isn't much treatment for it. You can get T shots, but those will not do more than slightly reduce developed mass.
I get the idea though that you are not too concerned whatever the cause since it helps your crossdressing.
One thing to check for is to feel carefully behind and around the nipples. If there is actual breast tissue then there could be something like the above going on. Most of the time, man boobs do not have well developed breast tissue (though your mileage may vary (YMMV), just the fatty deposits that make them rounder and more prominent, kinda droopy and soft enough to scoop into a bra cup and make a bit of cleavage.
Another thing you can do is stand in front of the mirror, and raise one hand over your head. As the flesh stretches out there may be a more solid though soft mass around the nipples, about 3 inches across for an A cup, larger for a B cup, etc., often with small nodules if your fingers are sensitive, with the off hand. That is likely breast tissue and not just man boob fat.
Men have small amounts of nodular breast buds (which is why men can take certain forms of hormones and lactate some), but if it is more than a silver dollar size at your age, have your hormone levels checked out. If low testosterone and/or higher than normal estrogen is present in your blood work, you may need to go ahead and have the genetic karyotype run. Your endocrinologist should know. If it is breast tissue, then you need to consider self exams for tumors periodically at least, and if breasts and tissue continue to develop, then mamograms.
At least so I was told in 1995. Good luck.