in /etc/ssh/sshd_config (on some systems it is simply /etc/sshd_config) add:

AllowUsers root

AllowUsers joe

Note that as soon as you enable one AllowUsers option, no users can login unless they are listed. In other words, despite the PermitRootLogin setting, if you do not set "AllowUsers root" and have "AllowUsers joe" in your configuration, root cannot log in even with a proper key.

http://www.techrepublic.com/blog/linux-and-open-source/specify-who-can-log-in-via-openssh/