What they saw looked like a star, and that is what Matthew is recording. It's doesn't mean Matthew was incorrect as he was simply relaying what he had learned.
An angel can look like a star, no?
Why not?
Scripture and tradition does not tell us. Maybe they realized it when it disappeared and then reappeared? Maybe when it took a turn and started acting like an unusual light in the sky? Maybe when it came to rest over the Child? Maybe they never realized it was an angel? We simply don't know. My guess is that they discovered early on this was not a "normal" light in the sky and was likely confirmed to them to be a supernatural light when they came upon the house where the Child was. In fact, some of the Patristic sources mention that this "star" was similar to the pillar of fire which led the Israelites through the deserts during the exodus. Not that it looked like a pillar, but had characteristics of the pillar, namely stopping when the caravan stopped, etc.
The important thing is that just because Scriptures do not necessarily and explicitly say that the magi ultimately realized the star to be an angel or supernatural phenomenon does not mean this was not indeed the case.
I also believe this must have been the case. There were plenty of amazing events and alignments occurring in the skies at that time, for example Jupiter, Saturn, Venus, the constellations, etc. There were plenty of signs to point to them that a King of Judah was to be born, the awaited Messiah. The angel/star which led them was an additional phenomenon which they witnessed and whose goal was specifically to bring these three wise mages to the Christ Child. Thus you have the celestial signs from the movements of the stars/planets, and then you have the supernatural manifestation of the angel/star guiding the seekers to the location of the Child.
That pictorial representation of Jupiter over Bethlehem does not point to the exact location of the child or even specific house. The Scriptures states that it came and stood over the Child, not over the city. Jupiter's position over Bethlehem is significant, but it is not the 'star' which is described in Scriptures.
The Scriptures described historical events more as a synoptic model. In developing the Nativity story, St. Luke's describes the events as they were happening and understood. To the magi, it did look like a star, albeit a strange and wonderous star. The end of of this journey of following the star leads to Christ, Who is the central figure of this story. That St. Luke does not explain if the magi ended up realizing that the star was indeed a messenger angel does not exclude this to actually be the case. After all, the crux of the narrative is not about the star or about the magi, but of the Christ Child.