If you havent see it already, check out the links in shor's comment below - there are some great resources in there. In some cases you can also consider surveying your current audience or customers through email, on-site surveys or SurveyMonkey. Be sure to ask for some profiling information that you can use for determining specific persona needs like age, sex, location, etc. (Probably best not to make it sound like a creepy text chat like I just did though...) :)
Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
Having an industry influencer publish a blog post on your site or turning an interview with them into a blog post can help to drive traffic both through organic search but also via that influencer promoting the content to their audience (see the backlinks section above). This can also help to add more variety to your content and show your visitors that you are active in your field.
Dig deep into everything you ever needed to know about links from anchor text to redirection. Read this series of pages to understand how and when to use nofollow and whether guest blogging is actually dead. If you're more into the link building side of things (working to improve the rankings on your site by earning links), go straight to the Beginner's Guide to Link Building.