Your husband is considering leaving you alone with your two children, both under 4 years old, because your first child is not affectionate towards him.
Just stop and think about this for a moment.
Does this sound to you like the behavior of a rational adult man? Not to me.
Your husband is not being “sensitive.” He is being mentally ill. You are enabling him to avoid getting help.