OPC is Cclub's new architecture for Org and Personal Colocated VMs. It's built using the Xen hypervisor and libvirt management layer. This approach gives both Computer Club administrators and the VM owners the ability to administer and manage the VMs, striking a balance between previous approaches we've taken. In this way, the Computer Club is able to easily handle the VMs when necessary (e.g., shutting VMs down for a planned power outage or because ISO alerts us that there was a break-in), but the owners are also able to utilize the management layer to directly solve problems (e.g., rebooting a crashed VM or using console access to debug a boot problem).

Basics

As mentioned above, OPC uses libvirt to handle VM management. There are a wide variety of tools available that use libvirt's tooling. The primary command line tools is virsh. You may already be familiar with virsh if you previously had a VM on the Daemon Engine architecture. Virsh is most easily available from the physical machines hosting OPC VMs. (Note, however, that the physical machines are on CMU-internal IP addresses. To connect to one of them via ssh, you will either need to connect from the CMU campus, or go through a machine at CMU with a public IP address as an intermediary.) The OPC physical machines are set up so that a virsh command without any --connect argument will access the local VMs.

Here are some example operations that you can perform. The commands will only show and work on VMs you have access to--VMs that you own, or VMs that are owned by a group you belong to. (In the example, I able to access two VMs: the stretch-test VM owned by the cclub group and the kbare.opc-test VM that I personally own.)

Disallowed Commands

Not all commands for managing VMs are available. As a general rule, commands that do not affect a VM's persistent configuration or resource allocation are allowed. Other commands are blocked.

E.g., 

kbare@opc-hp-01:~$ virsh setmem kbare:kbare.opc-test 20GB
error: access denied

Additional References

Remote Management

Libvirt supports remote management, allowing you to run virsh and other libvirt applications on your own machine in order to manage OPC VMs. For example, virt-manager is a GUI tool that supports many of the same operations as virsh. You can also run virt-viewer to connect to the graphical console of your VM.

We only support remote access via ssh tunneling, with xen+ssh:// URLs. E.g., you will need to use an option like --connect xen+ssh://opc-hp-01.club.cc.local.cmu.edu/ with the libvirt applications. Unfortunately, this requires additional configuration if you are not connecting from CMU. And, even if you are connecting from CMU, additional configuration is advisable for the smoothest operation.

SSH Configuration: Persistent Connections

Libvirt applications may need several session with the OPC physical machine in order to accomplish their work. This can slow things down significantly, as initial negotiations for an SSH connection require some time-consuming cryptography. For better responsiveness, we recommend setting up OpenSSH's support for shared persistent connections. To enable this for the OPC physical machines, add the following to the end of ~/.ssh/config: 

Host opc-*.club.cc.local.cmu.edu
ControlMaster auto
ControlPath ~/.ssh/control-%r-at-%h-%p
ControlPersist 15m

This instructs ssh to look for a special control socket before it attempts to connect to the OPC physical machines. If the control socket does not exist, ssh will connect normally, but will create the control socket and listen on it for commands. If the socket does exist, ssh will communicate over the socket to some existing, already-connected ssh process, telling it to open an additional channel for a new session on the target machine--which is quite fast. Moreover, the first ssh process will maintain its connection for 15 minutes after all sessions have been closed, meaning that if additional sessions are needed before the 15 minute timeout, they can also be created quickly.

SSH Configuration: Kerberos

Even if you set up persistent SSH connections, libvirt applications will still need to establish a new SSH connection if there isn't already one available. And this requires authentication, which would typically meaning typing in your Cclub password. This might be inconvenient, e.g., if you are trying to run a graphical application and SSH tries to read your password from a terminal. However, with some more configuration, you can make it so that ssh to Cclub system authenticates via Kerberos instead of asking for a password. For this to work, you may need to install a package containing the Kerberos clients. Then add the following directives to the end of ~/.ssh/config: 

Host *.club.cc.cmu.edu *.club.cc.local.cmu.edu
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes

Now if you get Kerberos tickets for your club account, ssh will not ask for your password when you connect to cclub machines, up until the point in time the tickets expire (by default 8 hours later). E.g., 

# Without Kerberos tickets, the ssh connection triggered by virsh prompts for a password
kbare@musicality:~$ virsh -c xen+ssh://opc-hp-01.club.cc.local.cmu.edu/ list --all
kbare@opc-hp-01.club.cc.local.cmu.edu's password: ^C

# XXX: Technically this should be 
# > kbare@musicality:~$ kinit kbare@CLUB.CC.CMU.EDU
# to hint at what needs to be done if the local user name is different than
# the club user name, and to support machines that are not set up to use
# CLUB.CC.CMU.EDU as their default realm.  However, it fails on my machine
# and I'm not really sure why. 
kbare@musicality:~$ kinit
kbare@CLUB.CC.CMU.EDU's Password: 

# After obtaining Kerberos tickets, there is no password prompt.
kbare@musicality:~$ virsh --connect xen+ssh://opc-hp-01.club.cc.local.cmu.edu/ list --all
 Id    Name                           State
----------------------------------------------------
 2     g:cclub:stretch-test           running
 -     kbare:kbare.opc-test           shut off

SSH Configuration: Access to OPC Machines from Off Campus

As mentioned, remote access to the OPC physical machines does not work by default if you are connecting from outside of CMU, since they have IP address that are only reachable from the CMU network. However, you can go through another Internet-accessible machine in order to reach them. OpenSSH can be configured to automatically set up and use a second SSH session as a tunnel when you request an ssh connection to any of the OPC physical machines. This is done by adding the following to the end of ~/.ssh/config: 

Host opc-*.club.cc.local.cmu.edu
ProxyCommand ssh -l %r -e none oyster.club.cc.cmu.edu nc -w 3 %h %p

XXX: Should not use oyster, as that will break when the current shells are replaced. Unfortunately, there are gremlins with the Kerberos handling for unix.club.cc.cmu.edu. Solution might be to create a special-purpose machine or CNAME, e.g., opc.club.cc.cmu.edu or opc-proxy.club.cc.cmu.edu.

When using this configuration, it is very helpful to also configure persistent connections and Kerberos, as recommended in the previous sections. Tunneled connections can take a lot longer to establish, making persistent connections a big time saver. And Kerberos eliminates the need to type in a password both for the underlying tunnel and for the connection through it to the OPC physical machine.

Access to a VM's Graphical Console

If you install virt-viewer on your machine, you can use the remote management functionality to access your VM's graphical console. This works by establishing an SSH session to the OPC physical machine and connecting to a VNC server connected to your VM.

Unfortunately, this isn't quite as smooth as it might be. To prevent unauthorized users from connecting to your VM, we've set up a VNC password. It's available through libvirt, but for some reason virt-viewer is not smart enough to find and use it. Consquently, you'll need to read it yourself using virsh dumpxml <vmname> --security-info. E.g., 

kbare@musicality:~$ virsh --connect xen+ssh://opc-hp-01.club.cc.local.cmu.edu/ dumpxml g:cclub:stretch-test --security-info | grep -i vnc.*pass
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' keymap='en-us' passwd='abCDefGH'>

The password will be shown as the passwd attribute of the graphics XML element. In this example, it is abCDefGH. The VNC password will not change, so you are welcome to memorize it or put it in a password manager, so that you can avoid the dumpxml step in the future.

Then, to connect to the graphics console, run virt-viewer --connect <url> <vmname> and type the VNC password into the pop-up password dialog. The program may grab your mouse. By default, the mouse can be released by pressing the left Ctrl and Alt keys simultaneously.

For more information on virt-viewer, see:

GUI Management of VMs

If you're not a big fan of command line tools, virt-manager provides a graphical interface to much of virsh's functionality.

When you first start virt-manager, you'll probably see an error message similar to the following:

Don't worry. It's trying to connect to VMs on your local machine, and you probably don't have any. To access your OPC VMs, you'll need to tell virt-manager how to connect to the OPC physical machine hosting them. To do this, go to "File" -> "Add Connection" and fill out the dialog. Remember, we use the Xen hypervisor and SSH for connections. For example:

After the connection completes, virt-manager will show the connection and list of your VMs on the physical machine.

To get more details and controls to manage a specific VM, you can double click on its entry in the list. This will open a new window for that VM. By default it will show a tab for the VM's graphical console, but, as with virt-viewer, it needs you to type in the VNC password.

As with virsh, the operations you are allowed to perform are restricted. You cannot create new VMs, change a VM's persistent configuration, or change resource allocations. E.g., you cannot increase the number of virtual CPUs allocated to your VM.

For more information on virt-manager, see: