What you describe is not a stupid consideration. Trunks are the absolute bare-bones starting point for a costume, and when a person wears more than that for a long time, that look becomes part of their trademark.
To abandon that trademark and return to what you seem to view as a kind of "purity" would also completely return the character to square one in terms of his visual design (except his hair and beard).
In design, you should always build on, rather than destroying, what came before, and this is a case where that is possible.