I think that was most likely the issue, because since I have been doing more regular (and smaller) water changes, I can see quite a difference in how well the male seems to feel. He is eating really well now, and I have even seen him picking up some of the Hikari "micro pellets" now that I also tried submerging a pinch so that it all sinks in one area instead of spreading all over the whole tank (and into the filter)
He is kind of a bum though, some mornings he chases the female around the tank constantly like he wants her out of his territory. Then some days, like today, they will happily pick around the tank eating together, and it seems like they get along fine. I really like these fish though, for that reason, they have so much character compared to the other fish I have kept.
edit: also I should have thought of the nitrate thing sooner too, but I guess since I have gotten away with 40+ppm in the past with fish like guppies and other "beginner" fish, I just didn't think to worry about it. It was definitely good for the plants, but they are still all doing fine with lower nitrates, just not growing as quickly!