Yeah, obscure port, maybe some security through obscurity with a relatively complex password (or shared ssh key).
If it's a Linux system that has other user accounts it's possible to restrict logins on those accounts to certain IP ranges (LAN IP range). Leaving only your "secure" account for outside access.
EDITED: 10 Dec 2014 20:26 by ANT_THOMAS