Was Joseph Really Suspicious of Mary's Pregnancy?

"“Now the birth of Jesus Christ took place in this way. When his mother Mary had been betrothed to Joseph, before they came together she was found to be with child of the Holy Spirit; 19 and her husband Joseph, being a just man and unwilling to put her to shame, resolved to send her away quietly. 20 But as he considered this, behold, an angel of the Lord appeared to him in a dream, saying, “Joseph, son of David, do not fear to take Mary your wife, for that which is conceived in her is of the Holy Spirit; 21 she will bear a son, and you shall call his name Jesus, for he will save his people from their sins”