I addressed a female colleague by this honorific today and was told it was "terrible" (not by her but by another lady who was listening.). I call women 'ma'am' all the time. (Women who I know, not strangers.) I see it as a mildly humorous term of respect. I wouldn't mind anyone calling me 'sir'-- in fact, I love it.
I see on the internet, having looked it up just now, that 'ma'am' is disliked because it implies the lady so addressed is old. Sheesh. It has nothing to do with age.
I think it's sad how the little courtesies of life become so fraught.