As a general rule, you're absolutely right - the one button blazer is a more casual style, and the fact that it is suede emphasizes that rule. To me, this is more of a weekend-type jacket, or something for casual Friday's.
I would pair such a jacket with a crisp striped button-down top or silky tie-front blouse and darker toned wide-leg trousers for the casual Friday look. Add platform Mary Janes or wedges, a leather bag (NOT suede) and a funky necklace and you are good to go.
For the weekend, I would shoot for more of a sophisticated-yet-casual look by pairing this jacket with a sleek turtleneck and a great pair of figure flattering designer jeans. Finish with funky open toe wedges or even a great pair of boots, an excellent handbag and a pair of dangles and you are absolutely ready for wheels-up to the Hamptons!
One thing to remember about suede: any other leather that you wear with it should NOT be suede. Patent leather or anything with a rich burnished finish is perfection.