Male produces two types of gametes (Sperms), 50% of sperms carry X chromosome, while the other 50% carry Y chromosome, while female produces only one type of gametes (ova), each carrying X chromosome. after fusion of the male and female gametes, the zygote would carry either XX or XY depending on whether the sperm carrying X or Y fertilized the ovum.
The zygote carrying XX would develop into a female baby and XY would form a male baby. Thus, the sex of the baby is determined by the father and not by the mother.