Yeah, obscure port, maybe some security through obscurity with a relatively complex password (or shared ssh key).
If it's a Linux system that has other user accounts it's possible to restrict logins on those accounts to certain IP ranges (LAN IP range). Leaving only your "secure" account for outside access.