Barring the two groups who are notorious for being quite a challenge to handle (those being children and animals) in my experience working to get the best out of your talent just takes a little common sense and patience.
Know your Stuff
The first thing I would say is to have a strong vision for what you want out of your actor’s performance. If you don’t have a clear idea of what you want to achieve with a scene, you won’t be able to provide any solid direction. Now this is not ALWAYS bad, as sometimes you can get a surprise performance out of your talent, but that will depend on your talent and how well they can pull something like that off. Usually it’s best to have a solid game plan in mind first, and if need be deviate from that as you shoot on set.
Don’t be Afraid to Experiment
When you’re on set you must also be willing to be flexible. A simple one-sentence line can be read in many different ways. Even if you have a good idea what it should sound like, play with it and let your talent experiment with it a bit…. time willing of course. They can sometimes come up with something that you hadn’t thought of before.
Also don’t be so rigid that you completely ignore all other input from either the crew or talent. If it’s reasonable, and again time willing, give it a try. At worst you can leave it on the preverbal cutting room floor if it doesn’t work out.
See it from their POV
Sometimes you need to put yourself in their shoes. It’s a whole different perspective from behind the lens than in front of the camera. Do a walk-through with them and discuss the tone, inflection, where they should be looking, how they should be moving and all of the other little tidbits that go into a performance. Once they have a good idea of what you want, and are comfortable with it, it will make the whole production go that much smoother.
Sometimes when you just aren’t getting the performance you want out of your talent, it can be very frustrating, but screaming at them or pulling your hair out (or theirs!) is not the route I would suggest. After a painful take, try “that was good, but why don’t we try it a little more like this”. When you make them feel good and comfortable in a situation you’re going to get better results than tearing someone’s head off. Small little nudges in the direction you want them to go can work wonders. You may have to shoot more takes, but in the end the results will be worth it.
I’ve also found that challenges can really help a performer as well. This goes especially double for more shy types and children. Again giving them encouragement but throwing in a “that was really good, but I know you have a great take in you, let’s try that again this way” can prove to be the decisive difference.
Great talent can be hard to find, especially in smaller markets or on gigs that don’t have Hollywood budgets, but if you as the director are willing to give a bit of coaching, a bit of nurturing and a healthy dose of guidance, you can make your production shine with their performance!