Was man made at the beginning or the end?

Matthew 19:4-5 (NKJV)

And He answered and said to them, "Have you not read that He who made them at the beginning 'made them male and female,' and said, 'For this reason a man shall leave his father and mother and be joined to his wife, and the two shall become one flesh'?

This is an interesting verse from the perspective of creation. Really, the context is about why divorce is wrong. You can see here an assumption Jesus had about his views on creation and the two chapters of Genesis. He says that man was made at the beginning. If there were billions of years of star and planet formation and then billions of years of evolution before man finally appears on the scene then Jesus should have said they were created near the end rather than at the beginning.

I believe the Bible to be the literal, inerrant, inspired word of God but when Jesus own words confirm it, that really settles the matter for me.