I wouldn't say it's been a constant downswing.
The Klitschko's were the worst thing to happen boxing. Not only for their style and dominance but because there was two of them and they wouldn't fight each other. Heavyweight boxing had always been the money division and they buried it. There still hasn't been an undisputed champion since Lennox Lewis which is insane.
There was an pretty big boost for boxing in the prime Mayweather/Pacquiao years but the delay and eventual disappointment of their fight turned a lot of people off. I'd say there's been another resurgence with he emergence of some top heavyweights while Canelo has been hugely popular. It's not what it was but it's damn sure better than it was in the mid 00's.
I actually MMA is in a bigger downswing than boxing right now. The UFC have pretty much followed the WWE model to a tee and are spamming content to appease their TV partners. Making more money but drawing less viewers. The star power of the fighters is dwindling every year.