Don’t bother with the adapter, just get drill bits with a hex base.

I’ve had the adapter for 20 years, and the hex-base bits for 10. Haven’t used the adapter since, my old bits sit in a box.

If your bits are sharp then using the impact for soft materials is fine. I find the impact doesn’t engage except for a moment at punch-through.