Is it rude to send back baby gifts?

I can see things from both points of view. You had no idea it would offend him, and he feels like you re-gifted something that was supposed to be significant to your children. Neither one of you is right or wrong, it's just a misunderstanding and had you known how much it meant to him, i'm sure you wouldn't have done it.