Those are just two exceptions to your statement. What you have said is a generalisation. And as we are discussing trans people, we are inherently NOT discussing the vast majority of the population, so I'm not sure why you would ignore any concept because it only applies to a minority.
Again, xy individuals are not always phenotypically male, and xx individuals are not always phenotypically female, despite it being the most common outcome. There is a specific gene on the y chromosome that, when expressed, is translated into a regulatory protein which kicks off a whole bunch of proteins which create a male phenotype out of the 'default' human body (female). This is what creates all the other phenotypes you would normally associate with maleness but mainly, at early human development stage, it just kickstarts the development of the male gonads and genitals.
The rest of the changes you mention really just occur at puberty, when other genetic signals initiate another massive release of hormones such as oestrogen and testosterone (though not exclusively) which shapes things like skeletal structure, muscle mass etc. What's important to remember that these things create a difference of DEGREE not KIND. Even the testicles and *smile* are essentially the same structures as the clitoris and ovaries which have been shaped differently by hormones released in different quantities at different times.
This is where you get a spectrum of human beings. Not the binary system of one or the other. Because it's not a switch of male or female, it's a release of a certain amount of femaleness or maleness. Weird example, there's a tribe of people in the carribean (I think) where a significant percentage of their kids begin life looking phenotypically female because the initial release of male determining hormones is not detected as efficiently by the cells of the sexual organs. As a result, they spend their early lives (up until puberty) looking like biological females, then all of a sudden, with the big second release of the biologically encoded hormones, they develop some very male characteristics, including male looking genitalia.
In reality, none of this describes transgendered people very well. This is more about intersex individuals - those who don't biologically fit the binary male/female system. But the spectrum of humans highlights that biological sex, as well as associated gender, is not a simple situation. I would think most people could understand that if sometimes your biology is in conflict with itself, your psychology can be in conflict with your biology.
There are also always going to be some biological females who find it easier to put on strength and size than some biological males, because, again, it's not as simple as you make it.