I read the memo and I don't remember him advocating one colour of the human skin over another. He was implying that women are worse at some stuff than men. His specific arguments were questionable, I would agree. But once again, I don't remember him writing that, say, people of African descend are worse at computer science than those of paler persuasion...