I don’t have any book recommendations but I can’t help but feel like the entire approach you’re trying to take might be too over generalized and you’re better off trying to approach each problematic social encounter one by one.

If for example you have family who’s down on their luck and trying to move into your living space despite you not wanting that, you need to consider what their other options are for living alone and if that would result in a quality of life you would be able to accept yourself, and weigh that against your own expectations for how the living situation would pan out in your mind.

Stuff like that family member’s previous behaviour, ability to show gratitude and value you equivalently, the degree to which they are responsible for their current living circumstances, etc, are all important to consider. This is nothing generalized advice about “boundaries” could possibly help with imo.

If on the other hand you’re a woman and have issues with men hitting on you at work, you have a completely different set of considerations you must make, with virtually no overlap with the previous example.