Aw, it's so hard when you and Hubby don't agree on baby care - especially hard when it's your first & you're tired & all!
I discovered babywearing when my first son was about 8 months old, and so when my daughter (now 8 months old) was a newborn, I wore her in a stretchy wrap ALL the time - she took her morning naps in the wrap, and during dinner time, and anytime we went anywhere in public, she would hang out in there - and she was happy! I don't think that you can wear your baby too much, but I know you don't want to cause friction between you and your husband either - if your baby is just as happy sleeping in her crib, there's nothing wrong with that either. One thing, though - after my daughter was a couple of months old, we transitioned her to taking all of her naps in her crib, and now she won't really sleep anywhere else (except driving in the car if she's really tired) - she used to sleep in the wrap so well, it was so nice when we were out, she would just fuss a little, and fall asleep in the wrap - but now she won't and she gets REALLY fussy if we're out and she's tired! But you have to do whatever works for your family.
Sorry, not really giving any 'answers', I know!