Bhyve overview – Episode 2 vmrc

In article bhyve overview – Episode 1 we have seen that bhyve is interesting but the script is too limited to be able to have fun. Let’s see if vmrc we offers more possibilities or not.
Vmrc installation

Let’s see if vmrc is available in the repositories pkgng:

# pkg search vmrc

ARF not :(! And obviously it is not more in ports. You’ll need to install it by hand. Okay, Let’s install git:

# pkg install git

Then grab the sources of the project:

# git clone

Enter the directory and install everything using the script install – and install –

# cd vmrc
# cd templates

The vmrc documentation recommends sysutils/grub2-bhyve for Linux guest:

# pkg install grub2-bhyve

Using vmrc
Creation of a FreeBSD guest

The vmrc directory recovered by git (/ root/vmrc in our example) in the script that allows to choose the type of vm needed and automate the creation and installation. Run

# cd/root/vmrc

We can then choose the type of VM we want to create:

Listing templates in/usr/local/vmrc/templates /.

t_centos65 t_freebsd92stablezfs t_pfsense –
t_freebsd10 t_freebsd92zfs t_ubuntu1304
t_freebsd10malloc t_freenas t_ubuntu1310
t_freebsd10zfs t_freenas – virtio21pf
t_freebsd11current t_master_template virtio83
t_freebsd11currentzfs t_openbsd virtio83pf
t_freebsd92 t_openbsd – virtio84
t_freebsd92stable t_pfsense virtio91

Enter a template to use:

Here as I want a classic Freebsd VM I selected t_freebsd10 and validated. Then we give a name to the vm, I put freebsd.

Enter a custom name without ID or leave blank to keep vm2

The script will create a virtual drive by 2G then download different sets (base.txz and kernel.txz) of FreeBSD to install. Now, let’s see if you can boot. For this we will use the vm service that must be enable in the /etc/rc.conf.local:
vm_enable = “YES”

And then it starts the vm:

# vm start freebsd service
VM: starting freebsd
VM: freebsd: No. configuration file found: Skipping…

Wow! That doesn’t work! Strange. Let’s take a look at the/usr/local/vmrc/vm:

# ls/usr/local/vmrc/vm.
freebsd2 openbsd01 VM

It appears he decided to appoint my freebsd2 vm. I feel come confusion when we begin to have a lot of vm, but maybe that I would have the choose another name. Okay, either use freebsd2:

# vm start freebsd2 service
VM: starting freebsd2
starting freebsd2
Entering f_load()
Entering f_eptcheck()
Entering f_vmmcheck()
f_vmmcheck: vmm.ko kernel module not loaded. Loading…
f_netstart: bridgestp.ko kernel module not loaded. Loading…
f_netstart: if_bridge.ko kernel module not loaded. Loading…
f_netstart: if_tap.ko kernel module not loaded. Loading… 0-> 1
Creating the bridge0 network interface.
Associating em0 with bridge0.
Running: ifconfig bridge0 addm em0 up
f_load: Attaching raw image /usr/local/vmrc/vm//freebsd2/freebsd2.img for fsck
f_load: DEBUG: disk image is usr/local/vmrc/vm//freebsd2/freebsd2.img; vm_device is md1
f_load: fsck_ufs y md1p2
* / dev/md1p2
* Last Mounted on/usr/local/vmrc/vm/freebsd2/mnt
* Phase 1 – Check Blocks and Sizes
* Phase 2 – Check Pathnames
* Phase 3 – Check Connectivity
* Phase 4 – Check Reference Counts
* Phase 5 – Check Cyl groups
13303 files, 139007 used, 368296 free (208 frags, 46011 blocks, 0.0% fragmentation)

Entering f_mddestroy()
f_mddestroy: Destroying all memory devices associated with freebsd2
f_mddestroy: Destroying mdconfig device: md0
mdconfig: ioctl(/dev/mdctl): Device busy
f_mddestroy: Destroying mdconfig device: md1
f_load: Running the bhyveload command:
/ usr/sbin/bhyveload m 1024-d /usr/local/vmrc/vm//freebsd2/freebsd2.img freebsd2
Consoles: userboot

FreeBSD/amd64 User boot, Revision 1.1
(, Thu Jan 16 22:18:02 UTC 2014)
Loading /boot/defaults/loader.conf
can’t find ‘kernel’

can’t load ‘kernel’

Type ‘?’ for a list of commands, ‘help’ for more detailed help.

Okay, it doesn’t always work, however I note the following points:

VMRC will mount if necessary modules, such as vmm
VMRC check the filesystem of the virtual disk before mounting
VMRC created a bridge on the host, and it is cool

Okay, let’s change TTY (ctrl + alt + f2 or alt + f2 for vmware) and stop the vm:

vm stop service freebsd2

Then go back into tty1 and let’s see what happened. Analyze the contents of/usr/local/vmrc/vm/freebsd2/mnt to see if the kernel is actually missing (we note in passing that vmrc has the delicacy to a mount point, to see what there is in the freebsd2.img file from the host):

# ls/usr/local/vmrc/vm/freebsd2/mnt /.
.cshrc bin lib proc sys
.profile boot libexec rescue tmp
.snap dev media root usr

Hmm, at first glance everything seems correct, but it seems that a kernel file is missing in boot/kernel.

Do another virtual machine to see, which we call freetest
#. /mkvm

Good note that he named freetest3. No importance, let’s start this VM

# vm start freetest3 service
______ ____ _____ _____
| ____| | _ \ / ____| __ \
| |___ _ __ ___ ___ | |_) | (___ | | | |
| ___| ‘__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___|| ____/|_____/|_____/
s …..—…….–. -/
+ - Welcome to FreeBSD-+ + o.-' y ' +.
| | Yo'. :o
| 1 boot Multi User [Enter] | y/ -/ -o/
| 2 boot [S] ingle User |.-: / sy +:.
| 3. [Esc] ape to prompt loader | /
— /
| 4 reboot | : :
| | : :
| Options: | / /
| 5 configure Boot [O] options… | .- -.
| | — -.
| | : :
| | .– `–.
| | .—…..—-.

f_load: freetest3 appears to have loaded.
Entering f_boot()
f_boot: Starting VM Networking. “File exists” warnings are okay.
Running: ifconfig tap8030 create
ifconfig: create: bad value
Running: ifconfig bridge0 addm tap8030 up
ifconfig: BRDGADD tap8030: File exists
Checking for preflight script/usr/local/vmrc/vm / / freetest3/freetest3.

f_boot: Running the bhyveload command:
/ usr/sbin/bhyveload m 1024-d /usr/local/vmrc/vm//freetest3/freetest3.img freetest3
f_boot: nmdm.ko is loaded.
f_boot: Booting freetest3 on console/dev/nmdm3A
root@FreeBSD:~/VMRC #

This time, it works. So for some unknown reason, the last time the installation of the kernel does not successfully completed.

Let’s get to the freetest3 console:

# vm attach freetest3 service

One sees then all the boot sequence, then we arrive at the login:

Thu Jul 24 07:21:19 PDT 2014

FreeBSD/amd64 (bhyve) (console)


Silly question: what is the root password defined by mkvm? Apparently is not root, it is not empty, it is not the name of the vm. And the instructions say nothing about it. Turn on tty2 (ctrl + alt + f2 or alt + f2) and let’s take a look at the script itself:

# cat/usr/local/vmrc/templates/t_freebsd10. grep password
vm_password = “bsd” # VM password (clear text for now) (FreeBSD only)

This is! The root password is bsd!

Let’s see if the internet connection works:

# ping

Hum! Apparently not! Let’s see why:

# cat /etc/rc.conf
ifconfig_vtnet0 = “ netmask”
defaultrouter = “”

Hum, this subnet does not match the network host (, so unless there is of NAT that can not walk as easily. On the host, the command pfctl – sr tells me that pf does not work. So no NAT. tokens a look at the /usr/local/vmrc/vm/freetest3/freetest3.conf:

vm_ipv4 = “” # VM IPv4 address (blank for DHCP) (FreeBSD only)
vm_gw = “” # VM IPv4 gateway (FreeBSD only)

So let’s change this to specify an address within the same range as the host, based on the principle that vmrc activates a bridge:

vm_ipv4 = “” # VM IPv4 address (blank for DHCP) (FreeBSD only)
vm_gw = “” # VM IPv4 gateway (FreeBSD only)

Then reboot the VM:

# vm service restart freetest3

The VM-enter and modify its /etc/rc.conf:

ifconfig_vtnet0 = “ netmask”
defaultrouter = “”

Are relaunching the network:

# service netif restart

Now launch a ping to the internet:

# ping

Hum, even once, that doesn’t work. Try a ping to, the vmware gateway:

# ping
PING ( 56 data bytes
64 bytes from icmp_seq = 0 ttl = 128 time = 2.127 ms

This time it works!

So in my case no access to the internet, but I have the following hypothesis: it is VMware that denies packets coming from my bridge.

The balance sheet is already much more positive than with We managed to install – very easily – and operate a guest FreeBSD10. We were unable to use the network, but we were not far away.
Guest Linux

Let’s take our ISO of debian, debian – 7.6.0 – amd64 – netinst.iso, and see if vmrc will succeed to the boot and install a debian guest system. Let’s take a look at the templates:

# ls/usr/local/vmrc/templates /.
t_centos65 t_freebsd92stablezfs t_pfsense –
t_freebsd10 t_freebsd92zfs t_ubuntu1304
t_freebsd10malloc t_freenas t_ubuntu1310
t_freebsd10zfs t_freenas – virtio21pf
t_freebsd11current t_master_template virtio83
t_freebsd11currentzfs t_openbsd virtio83pf
t_freebsd92 t_openbsd – virtio84
t_freebsd92stable t_pfsense virtio91

No template for debian :(but as I am crazy, I’ll take the t_ubuntu1304 template and modify it to Debian. I begin by duplicate it:

# cd/usr/local/vmrc/templates
# cp t_ubuntu1304 t_debian76

Then I edit my template:

# My awesome debian template

vm_cpus = “1” # Number of VM virtual CPUs (max 16)
vm_ram = “256” # VM RAM Allocation in MB
vm_console = “nmdm” # stdio, nmdm, tmux gold tmux-detached (sysutils/tmux)
vm_os_type = “linux” # freebsd, openbsd, or linux
vm_os_ver = “debian7.6” # Exact OS version if auto-fetching
vm_dev_layout = “gpt” # “gpt” gold “mbr” volume layout (FreeBSD only)
vm_dev_type = “img” # ‘img’ for image, “zvol” or blank for other device
vm_device = “” # An existing device (no udev) i.e. “ada1”
vm_dev_size = “10G” # M or G for raw “img” volumes
vm_hostname = “” # VM hostname (FreeBSD only)
vm_timezone = “” # VM timezone (FreeBSD only)
vm_ipv4_addr = “” # Experimental for jail use
vm_searchdomain = “” # (FreeBSD only) Commented out below
vm_dns_addr = “” # (FreeBSD only) Commented out below
dist_site = “” # Hostname and directory for binary distribution sets
iso_site = “”
# Hostname and directory for ISO image
iso_img = “debian – 7.6.0 – amd64 – netinst.iso”
# ISO filename for remote fetch
grub_boot_cmd = “/ usr/local/sbin/grub-bhyve r hd0, msdos1 m ${host_vmroot}/${vm_name}/ m $vm_ ‘
RAM $vm_name”
# grub – bhyve command to boot from IMG
grub_iso_cmd = “/ usr/local/sbin/grub-bhyve r cd0 m ${host_vmroot}/${vm_name}/ m $vm_ram $vm_”
# grub – bhyve command to boot from ISO
vm_hostbridge = “” # “amd_” for the AMD hostbridge
bhyve_flags = “” # Additional bhyve (8) flags
virtio_type = “virtio-blk” # “ahci-hd” or “virtio-blk”

Then it starts the creation of the vm I’ll appoint debian:

# cd/root/vmrc

During the creation of the VM, I get lots of errors of this type:

No. such file or directory

I note that the template seems to behave as in a FreeBSD guest because I see pass the loader.conf. Same problem if I use ubuntu provided templates. Are they bugues? Okay, the documentation says after provisioned vm, to use a specific command to start because it uses an iso. We’ll assume that the provisioning is done and run the command:

Clear All
# /usr/local/etc/rc.d/vm iso debian0
VM: booting the ISO for VM debian0
Entering f_grubcheck()
Entering f_tmuxcheck()
Entering f_eptcheck()
Entering f_vmmcheck()
f_vmmcheck: vmm.ko is loaded.
f_netstart: bridgestp.ko is loaded.
f_netstart: if_bridge.ko is loaded.
f_netstart: if_tap.ko is loaded.
f_netstart: bridge0 exists.
Member: em0 flags = 143
f_netstart: Hussein is assoc. with host0.
Entering f_iso()
f_iso: /usr/local/vmrc/vm//debian0/debian0.img found.
f_iso: Checking for /usr/local/vmrc/distributions//debian7.6/
f_iso: Checking for /usr/local/vmrc/vm//debian0/debian0.iso
f_iso: Fetching debian – 7.6.0 – amd64 – netinst.iso
/usr/local/VMRC/distributions//debian7.6//debi100% of 222 MB 2520 kBps 01m30s
f_iso: Copying debian – 7.6.0 – amd64 – netinst.iso to /usr/local/vmrc/vm//debian0/debian0.iso
Copying debian – 7.6.0 – amd64 – netinst.iso to /usr/local/vmrc/vm//debian0/debian0.iso
f_iso: Creating /usr/local/vmrc/vm//debian0/
(hd0) /usr/local/vmrc/vm//debian0/debian0.img
(cd0) /usr/local/vmrc/vm//debian0/debian0.iso
f_iso: Running the ISO grub command:
/ usr/local/sbin/grub-bhyve r cd0 m /usr/local/vmrc//debian0/ m 256 debian0
Could not create VM debian0
Error in initializing VM
funcname: Calling f_boot
Entering f_boot()
f_boot: debian0 is not loaded. Skipping…

Fail… again, no way to boot Linux. But I have the impression that this time the problem is localized on grub2-bhyve because vmrc has downloaded the iso and ordered the execution of the vm.

The article is long enough, we’ll stop there.
Crashes and incidents

I have observed several crashes or strange behaviour with FreeBSD. I met the system freezes, but the most annoying and most common bug is auto-startup of the vm. Indeed, since vm_enable = “YES” has been added to the rc.conf.local, VMS are launched at startup of the host. In any case it does not matter, except that the vm launch in the foreground, and systematically block on the bootloader. At this stage no keyboard typing cannot be taken into account. It must therefore restart FreeBSD directly in the vmware console and press CTRL + c at boot to cancel the launch of the vm. It is a critical and very annoying bug that removes any possibility of using vmrc in production at the moment. I also encountered another strange problem: the root password of the host who jumps. Sometimes, after a reboot or the launch of a vm, I am unable of me logger as root on the host, either live or SSH, because the password is rejected. I have no alternative but to reset me or load a VMware snapshot to go back.

VMRC brings much compared to the script as it allows to automate the installation of VMS, after the manner of lxc or ezjail, which facilitates us enough life. However only FreeBSD templates seem to be functional and I still couldn’t start a Linux guest. No access to the network no longer but I draw no conclusions on this point. Finally we feel the experimental side of vmrc with critical bugs at startup of the host, preventing any use in production at the moment. I try to contact the author for vmrc to get help on this point. VMRC is an interesting to follow project.