A cardholder goes in and out of a pocket dozens of times a day, which means it picks up more oil, lint, and grit than most other accessories. Most of the wear people blame on "cheap materials" is actually just accumulated dirt that was never cleaned off.
Wipe it down weekly with a dry microfiber cloth, and for leather, a small amount of leather conditioner every few months keeps the material from drying out and cracking at the fold. For fabric or synthetic shells, a barely-damp cloth handles most marks without soaking the stitching.
Avoid leaving it in direct sun for long stretches - on a dashboard or a windowsill - since UV exposure fades and dries out most materials faster than daily handling ever will.
