Hi Chunk and welcome to the forum
It is not an easy fix, and yes the whole assembly would have to be removed and replaced.
Can you not look at your repair again and try to fix it (better), and other than that have a look at scrapyards on-line for the whole assembly, as that would be your cheapest option. You might find it on Ebay as well. Good luck.
If its too much work , just glue the facia of the sunglass holder flat to the roof, and do not use it, and put your sunglasses elsewhere