You can grant friends access to your computer with permissions set specifically to just the controller, the mouse, the keyboard or some combination of the three. You can change these settings in your Friend's Page to permanently manage your friend's permission.
On each connection, you can change the permissions for the friend during that connection via the hosting sidebar. In this case, Legolas has permissions for just the controller.
If someone is joining the gaming session as a friend, they will have the permissions you granted by default. If they're joining through a link or via a Party, they will not have permission to use any input device. You must grant them access to the input devices you'd like to provide in the hosting sidebar.