It's easy for us to want to curse Ishmael. Actually, to want to curse Abraham - if he could have just listened to God and waited like he was supposed to, none of this would have happened.
If he could have controlled himself and trusted God, he wouldn't have taken Sarah's slave woman to bed, wouldn't have had a child through her, wouldn't have had to cast them into the desert, and wouldn't have made it possible for his great-grandson to be sold into slavery to the great-grandsons on the other side of his affair.
See? This is what happens when you sin. This is exactly what happens when you sin.
That's how our minds think, anyway.
No sin -> no Ishamel -> no Ishmaelite traders -> no slavery in Egypt.
But let's be real: that famine was coming whether Joseph was there or not.
The famine was coming on the land. The people were going to be starving. Jacob and his sons were going somewhere looking for food; they were ready to give everything they had to someone. While it's true that if Joseph were not in Egypt with his powerful gift of interpreting dreams, Egypt would not have had the storehouses that they did and would not have been the place to be able to provide for the rest of the world, that wouldn't have stopped Joseph's brothers from scouring the world for help that might be anywhere.
And it's quite possible that without Joseph in Egypt, there wouldn't have been help anywhere and the whole world would have just starved to death.
It's true that God didn't need Joseph in Egypt to save Jacob and his sons; He sent a raven with food to Elijah. He provided manna for His people in the wilderness. Just as He drew a line between Israel and Egypt in the course of the plagues, He could have drawn a line between them in the famine. But then, the rest of the peoples would have died. God needed Joseph in Egypt to save the world - to save everyone, not just His chosen people.
We also have to acknowledge that if Joseph never goes to Egypt, his brothers never go to Egypt (probably). If his brothers never go to Egypt, God's people are not enslaved in Egypt. And we might think that's a good thing - that's the argument we laid out above. But we have to keep going here.
Because if there's no slavery in Egypt, there's no show of God's power in Egypt, either. There's no exodus and parting of the Red Sea. There's no provision in the wilderness. There's no deliverance into the Promised Land.
Think about this - if there's no show of God's power in Egypt, there's no Passover. If there's no Passover, there's no sacrifice, no lamb. If there's no lamb, what kind of sense does Jesus make?
Do you see how it's not so simple as to say that things would have been "better" without Ishmael? It's easy for us to look at it and say that without Ishmael, these bad things don't happen, but it's not that simple. Bad things still happen. But without Ishmael, there's no glory.
(We could say the same thing about Judas Iscariot, by the way. Without a betrayal, there's no cross; but without a cross, there's no salvation.)