Under 20 liters, every item has to justify itself twice: once for whether you'll use it, and once for how much room it takes relative to how often you'll use it.
A change of clothes, a slim toiletry kit, a phone charger, and a compact rain layer cover almost every two-day trip. Anything beyond that - a second pair of shoes, a full-size toiletry bottle - is usually the first thing to get left behind once the bag stops zipping.
The habit worth building isn't packing less overall, it's packing the same tight list every time so you stop re-deciding what's essential on every trip.