Start the virtual machine with:

VBoxHeadless -s  <vmname>

If the RDP server is working well then as an output from the previous command will be:

Starting virtual machine: VRDE server is listening on port 3389.

If you get an error than:

1. Install relevant extension pack
Check the version of VirtualBox you are running

VBoxManage -v

download the relevant extension pack from VirtualBox site and install it:

sudo VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-x.y.z.w.vbox-extpack

– Enable the VRDE (VirtualBox Remote Desktop Extension)

VBoxManage modifyvm “Ubuntu Server 16.04.5 v2” –vrde on

– Connect remotely to the VirtualBox vm
rdesktop -N <vbox_interface_ip_address>

You may need to install rdesktop if you don’t already have it.

If you get and error then add the address by:

VBoxManage modifyvm <vm_name> –vrdeaddress <vbox_interface_ip_address>

For <vbox_interface_ip_address>  you select the IP of VBox interface that you get with: ifconfig. For example if you are using hostonly network it is the IP of vboxnet0 (e.g. 192.168.56.1).

Additionally it may help:

VBoxManage setproperty vrdeextpack “Oracle VM VirtualBox Extension Pack”