In the wild, adding a friend means allowing them to build within 3 chunks of your land. If you add to many, it could cause some problems later on when trying to build and expand. A remedy to this is to just have the other person you want to build with just build somewhere near by, or on the other side of your two friends (just have them add that person as a friend.)
In the towny world, you can add an unlimited number of friends. Meaning they can build, destroy, ect... on your plots. /res friend add [Name] . If you have more than 4 or 5 people wanting to build together, you may want to consider starting a small town or even joining a town that will allow you to have plots together.
Either way, here are two options to help you get around the limited wilderness friend additions. Keep in mind if you're wanting to allow someone to build and destroy on your wilderness plots, this is also possible with /wild builder.