Virtual machines are often used by security researchers to sandbox malware samples for analysis, or to protect a machine from a potentially hazardous activity. The theory is that any security threat or malicious behaviour will be restricted to the virtual environment which can be discarded and then restored to pristine condition after use.
Virtual machines are sometimes thought of as impenetrable barriers between the guest and host, but in reality they're (usually) just another layer of software between you and the attacker. As with any complex application, it would be naive to think such a large codebase could be written without some serious bugs creeping in. If any of those bugs are exploitable, attackers restricted to the guest could potentially break out onto the host machine.
investigated this topic and presented a paper at CanSecWest on a number of ways that an attacker could break out of a virtual machine.
Most of the attacks identified were flaws, such as buffer overflows, in emulated hardware devices. One example of this is missing bounds checking in bitblt routines, which are used for moving rectangular blocks of data around the display. If exploited, by specifying pathological parameters for the operation, this could lead to an attacker compromising the virtual machine process.
While you would typically require root (or equivalent) privileges in the guest to interact with a device at the low level required, device drivers will often offload the parameter checking required onto the hardware, so in theory an unprivileged attacker could be able to access flaws like this by simply interacting with the regular API or system call interface provided by the guest operating system.