Accessing USB devices from Virtualbox Windows guest on Linux host

This is one issue that had troubled me for long but was not important enough for me to put any effort into soving it. All this changed when I had to upgrade my BIOS on my Dell Optiplex 9020 machine. The problems are:

  • One cannot upgrade BIOS from a Linux machine, a Windows installation is a must. At least, a DOS bootable CD, USB Flash drive is needed
  • I run Manjaro on this machine and installing Windows just for the BIOS upgrade is just too much

So, I had to find a way to access USB devices on my Windows VM guest. I have never been able to do that. Primarily because, I never tried. This time I had to. So here is what is needed:

  1. Go to the Virtualbox Downloads page
  2. If the Virtualbox installed on the system is not the latest move to the Virtualbox Old Builds section
  3. Download the Extension PackVirtualboxWebsite_ExtensionPackLink
  4. Hit the link and the Firefox Save File Dialog pops up with Virtualbox selected as the app to open the file with. Just say OK:Firefox_SaveFileDialog_VirtualboxExtensionPack
  5. Virtualbox will try to install the Extension Pack. It will pop up the following EULA. Scroll to the bottom and the I Agree and I Disagree buttons will get enabled. You know what to choose:Virtualbox_ExtensionPack_EULA
  6. If all goes well, Virtualbox will throw this at you:Virtualbox_ExtensionPack_InstalledSuccessfully
  7. It is time for some radical changes in the system. Bring to life a terminal emulator and feed it the following command to add the desired user to the vboxusers group:
     usermod -a G vboxusers <User_Name>
  8. Now move to the settings of the Virtualbox Windows guest and click the USB settings. Make sure that you click on all checkboxes and finally the settings dialog looks like this:Virtualbox_USBSettings_AddFilter
  9. Make sure that the device that you want to access in the Windows guest is NOT mounted on the Linux host. If it is, unmount it. Then, click on the button with the USB icon which has the green colored plus (+) sign. The above picture shows the tooltip for the said button. Click on that and all available USB devices on the machine will be listed. Choose the one you want to add a filter for. Adding a filter will make the device available on the Windows guest. This makes the device completely inaccessible on the host. Every time you bootstrap the Windows guest, the filter will make sure that the device is available in the guest.
  10. Some driver installations will happen in the guest once the filter is added. A restart of the guest and host would be good for the society.

 

 

 

 

Advertisements

Unified Extensible Firmware Interface (UEFI) and Logical Volume Manager (LVM) adventures: How upgrading to UEFI resulted in this epic post

This is a work-in-progress…

It all started when I found my backup plan lacking in all ways imaginable. To be really honest, I never had one. This is not because I am digitally retarded or lazy (this is debatable) or lack resources. I did not have much to back up. Yes, in this age where digital asset accumulation is natural and people tend to have terabytes of data without even realizing, I had nothing worth a planned back-up:

  • The important documents, that are very few in number, have been sitting cosily in my mails
  • I, somehow, relied only on text based learning materials which were available online

In the past few months, I have seen myself learning a lot using videos – Mathematics, Computer Science, Arts etcetera. This, inevitably, means a growing digital wealth. One fine day, last week, I found that my 1 Tera Byte (TB) disk was more than 50% full. That is when I realized the folly in my ways.

I purchased a couple of the Transcend 2 TB Hard Drives:

I immediately backed-up all my digital belongings. With all my data safe, my brain started its creative endeavours. What was unthinkable became a real possibility now. Suddenly it occured to me:

  • That I was using Legacy Boot Options on my Dell Optiplex 9020 machine when UEFI was very much waiting for me
  • That the BIOS my machine was shipped with was version A01 while A07 was available on Dell’s website
  • That I was using the old partition scheme on my machine and had never experimented with Logical Volume Manager (LVM)
  • That Manjaro installation had some troubles as the GUI windows weren’t refreshing properly and making me mad (this was a minor inconvinience but humans are just insatiable)

Some History

When I installed Manjaro in the first week of October 2013, there was no UEFI or LVM support in the distro. They added these to the distro only in December 2013. I had very little idea as to how UEFI and LVM actually work and was not bothered to learn because I was already burdened with a lot of stuff. With no knowledge to bank on, using advanced features would have been foolish. So, I decided not to play with them.

Unified Extensible Firmware Interface Upgrade

This is quite easy. Just hit F2 after powering on the machine and the BIOS menu appears. Change the boot option to UEFI. Make sure that the Secure Boot option is not touhed at all. But, I knew that my BIOS version was very old and had to be upgraded compulsarily. This is a problem when you are on the Linux distro. This has nothing to do with Linux. This issue exists due to the attitude of large corporations towards FOSS.This is what Dells’s website instructs:

Installation instructions
BIOS Update Executable for Windows/DOS
Download
1. Click Download File to download the file.
2. When the File Download window appears, click Save to save the file to your hard drive.

Run the BIOS update utility from Windows environment
1. Browse to the location where you downloaded the file and double-click the new file.
2. Windows System will auto restart and update BIOS while system startup screen.
3. After BIOS update finished, system will auto reboot to take effect.

Run the BIOS update utility from DOS environment if Legacy Boot Mode(Non-Windows users).
1. Copy the downloaded file to a bootable DOS USB key.
2. Power on the system, then Press F12 key and Select "USB Storage Device" and Boot to DOS prompt.
3. Run the file by typing copied file name where the executable is located. 
4. DOS System will auto restart and update BIOS while system startup screen. 
5. After BIOS update finished, system will auto reboot to take effect.

Run the BIOS update utility from DOS environment if UEFI boot mode with Load Legacy Option disabled (Non-Windows users)
1. Copy the downloaded file to a bootable DOS USB key.
2. Power on the system, then go to BIOS Setup by pressing F2 and go to "General-Boot Sequence - Boot List Option".
3. Change "UEFI" to "Legacy" of Boot List Option.
4. Click "Apply","Exit" to save changes and reboot system.
5. Press F12, then Select "USB Storage Device" and Boot to DOS prompt.
6. Run the file by typing copied file name where the executable is located. 
7. DOS system will auto restart and update BIOS while system startup screen.
8. After BIOS update finished, system will auto reboot.
9. Go to BIOS Setup by pressing F2 and go to "General > Boot Sequence > Boot List Option".
10. Change "Legacy" to "UEFI" Boot Option.
11. Go to "Exit > Exit Save Changes" and reboot system.
--------------------------------------------------------------------------------
Note 1: You will need to provide a bootable DOS USB key. This executable file does not createthe DOS system files.
Note 2: Please make sure you suspend BitLocker encryption before updating BIOS on a BitLocker enabled system. 
If you don't enable BitLocker on your system you can ignore it.

You need Windows/DOS to flash the new version of the BIOS. Linux will NOT work. I had some choices, though:

  • Install Windows temporarily on the machine and flash the BIOS. Then, reinstall Manjaro. What happens if another update arrives in the near future? Will I be willing to install Windows temporarily and suffer the consequential pain? NO.
  • Use Virtualbox + Windows guest + Rufus or UNetBootin utility in Windows guest to create a bootable FreeDOS. This seems painless and an ideal way to handle the scenario.

The second option might not be clear to many. What is FreeDOS, some may ask. For that let us first see what DOS is:

DOS/dɒs/, short for Disk Operating System,[1] is an acronym for several closely related operating systems that dominated the IBM PC compatible market between 1981 and 1995, or until about 2000 including the partially DOS-based Microsoft Windows versions 95, 98, and Millennium Edition.

Related systems include MS-DOS, PC DOS, DR-DOS, FreeDOS, ROM-DOS, and PTS-DOS.

So, when Microsoft decided to kill DOS, many users were left in the lurch. FreeDOS was an attempt to help those users:

FreeDOS is a free DOS-compatible operating system that can be used to play games, run legacy software, or support embedded systems. FreeDOS is basically like the old MS-DOS, but better! For example, FreeDOS lets you access FAT32 file systems and use large disk support (LBA) — a feature not available in MS-DOS, and only included in Windows 95 and newer.

I knew I had to go the FreeDOS way and use Virtualbox to get a bootable FreeDOS USB. I decided to install latest Manjaro with UEFI enabled and then upgrading the BIOS.

So, I downloaded the latest Manjaro XFCE  edition with full support for UEFI and LVM. Then I did the following:

  • Used the dd command and created a bootable USB flash drive for Manjaro installation
  • Changed the Boot option from Legacy to UEFI in my BIOS settings (hit F2 after powering on the machine)
  • Started the Manjaro installation procedure

Manjaro Graphical Installation With UEFI and LVM Features

Manjaro’s graphical installer is a very beautifully crafted software. It resembles Ubuntu’s installer and is very practical and intuitive. The Installation type dialog had the options for an LVM partitioning in the automatic mode:

ManjaroGraphicalInstaller_InstallationType_LVM

and seemingly the LVM option is not available in the advanced mode:

ManjaroGraphicalInstaller_AdvancedInstallationMode

So, I chose automatic mode and let Manjaro installer choose the best partition scheme for me. There seems to be no way to modify the partition scheme to LVM in the automatic mode. This is limiting and would surely be addressed.

The lsblk command spits out this:

NAME                 MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                    8:0    0 931.5G  0 disk  
├─sda1                 8:1    0     2M  0 part  
├─sda2                 8:2    0   100M  0 part  /boot/efi
├─sda3                 8:3    0   256M  0 part  /boot
└─sda4                 8:4    0 931.2G  0 part  
  └─cryptManjaro     254:0    0 931.2G  0 crypt 
    ├─ManjaroVG-ManjaroRoot
                     254:1    0  29.3G  0 lvm   /
    ├─ManjaroVG-ManjaroHome
                     254:2    0 886.2G  0 lvm   /home
    └─ManjaroVG-ManjaroSwap
                     254:3    0  15.7G  0 lvm   [SWAP]

Evidently, my 1 TB hard drive (sda) was partitioned automatically into four partitions (sda1, sda2, sda3, sda4). The sda4 is used as an LVM that hosts the logical /, /home and SWAP partitions. I had chosen a separate partition for /home in the Installation type dialog’s automatic option. And yes, the LVM is LUKS encrypted:

GnomeDiskUtility_DiskLayout

Now, what exactly is LVM? This needs to be answered so as to understand whatever has been described in this section:

The full explanation of the concept would be quite a task. Thus, I resort to copying the important facts copied from the Ubuntu Wiki. It would be really benefitial for the reader to also read the following pages to understand LVM properly:

What is LVM?
LVM stands for Logical Volume Management. It is a system of managing logical volumes, or filesystems, that is much more advanced and flexible than the traditional method of partitioning a disk into one or more segments and formatting that partition with a filesystem.
Why use LVM?
For a long time I wondered why anyone would want to use LVM when you can use gparted to resize and move partitions just fine. The answer is that lvm can do these things better, and some nifty new things that you just can’t do otherwise. I will explain several tasks that lvm can do and why it does so better than other tools, then how to do them. First you should understand the basics of lvm.
The Basics
There are 3 concepts that LVM manages:
Volume Groups
Physical Volumes
Logical Volumes
A Volume Group is a named collection of physical and logical volumes. Typical systems only need one Volume Group to contain all of the physical and logical volumes on the system, and I like to name mine after the name of the machine. Physical Volumes correspond to disks; they are block devices that provide the space to store logical volumes. Logical volumes correspond to partitions: they hold a filesystem. Unlike partitions though, logical volumes get names rather than numbers, they can span across multiple disks, and do not have to be physically contiguous.

LVMComponents

Copied from http://www.markus-gattol.name/ws/lvm.html, Markus Gattol’s blog

[The five points below have been copied from the Ubuntu Wiki without modifications]

The Specifics
One of the biggest advantages LVM has is that most operations can be done on the fly, while the system is running. Most operations that you can do with gparted require that the partitions you are trying to manipulate are not in use at the time, so you have to boot from the livecd to perform them. You also often run into the limits of the msdos partition table format with gparted, including only 4 primary partitions, and all logical partitions must be contained within one contiguous extended partition.
Resizing Partitions
With gparted you can expand and shrink partitions, but only if they are not in use. LVM can expand a partition while it is mounted, if the filesystem used on it also supports that ( like the usual ext3/4 ). When expanding a partition, gparted can only expand it into adjacent free space, but LVM can use free space anywhere in the Volume Group, even on another disk. When using gparted, this restriction often means that you must move other partitions around to make space to expand one, which is a very time consuming process that can result in massive data loss if it fails or is interrupted ( power loss ).
Moving Partitions
Moving partitions with gparted is usually only necessary in the first place because of the requirement that partitions be physically contiguous, so you probably won’t ever need to do this with LVM. If you do, unlike gparted, LVM can move a partition while it is in use, and will not corrupt your data if it is interrupted. In the event that your system crashes or loses power during the move, you can simply restart it after rebooting and it will finish normally. When I got my SSD drive, I simply plugged it in, booted it up, and asked lvm to move my running root filesystem to the new drive in the background while I continued working. Another reason you might want to move is to replace an old disk with a new, larger one. You can migrate the system to the new disk while using it, and then remove the old one later.
Many Partitions
If you like to test various Linux distributions, or just different version of Ubuntu, or both, you can quickly end up with quite a few partitions. With conventional msdos partitions, this becomes problematic due to its limitations. With LVM you can create as many Logical Volumes as you wish, and it is usually quite easy since you usually have plenty of free space left. Usually people allocate the entire drive to one partition when they first install, but since extending a partition is so easy with LVM, there is no reason to do this. It is better to allocate only what you think you will need, and leave the rest of the space free for future use. If you end up running out of the initial allocation, adding more space to that volume is just one command that completes immediately while the system is running normally.
Snapshots
This is something you simply can not do without LVM. It allows you freeze an existing Logical Volume in time, at any moment, even while the system is running. You can continue to use the original volume normally, but the snapshot volume appears to be an image of the original, frozen in time at the moment you created it. You can use this to get a consistent filesystem image to back up, without shutting down the system. You can also use it to save the state of the system, so that you can later return to that state if you mess things up. You can even mount the snapshot volume and make changes to it, without affecting the original.

Manjaro Post-Installation

The installtion procedure of Manjaro is like most other established distros – as Zohan would put it, “Silky Smooth”. Once I was done with it, I was focussed on getting my machine updated. I fired up pacman and well, updated. Now I had Manjaro installed and updated. It is time to prepare for a BIOS update. How am I going to do it? The answer took me on a Virtualbox advenutre.

Once I came back from the Virtualbox adventure, I had with me a USB flash drive readily accessible form the Windows VM guest. I  used Rufus to create a bootable FreeDOS USB flash drive from the Windows guest and copied the BIOS executable that I downloaded from the Dell Website. I restarted my machine and hit F12 to get to the one-time boot menu. I chose the USB flash drive for booting and got into FreeDOS. This is where things got a bit confusing.

I had downloaded the following ISO from the official website. This is lacking in some features which makes it inappropriate for BIOS upgrade. Download this if you need to test or learn etc. but not for BIOS upgrade. For BIOS upgrade you need FULL CD:

FreeDOS 1.1 Base

fd11src.iso

Bittorrentor, via bittorrent

Features

  • FreeDOS 1.1 Base CD
  • Contains packages from BASE, and several useful utilities
  • Includes source code for everything
  • Install only, does not include LiveCD
  • 40 MB

The menu options available after the booting of this ISO are:

FreeDOS_OfficialCD_MenuOptions

So, I had to search for a full ISO image that could help me. Further net surfing helped me get my hands on this:

http://ftp.gwdg.de/pub/misc/freedos/files/distributions/1.0/fdfullcd.iso at http://www.z80.eu/freedoscd.html

There may be many other sources too! The only difference is that this ISO image is around 150 MB in size.

I created another FreeDOS bootable USB flash drive with this ISO and the menu options changed favourably. Choose the option highlighted in the image below:

FreeDOS_OfficialFULLCD_MenuOptions

Once the highlighted option is selected, the following appers:

FreeDOS_OfficialFULLCD_AfterFreeDOSSafeModeOption

With the command prompt at A:\>, the BIOS executable that I copied was not available. I used the dir command to look at the files. I had to change the working directory to C: drive. With that done, my file was available. Now, I could type the name of the file and hit enter to execute it. The executable takes control of the situation and flashes the new BIOS, A07 version. Done!

Conic Sections and the Double-Napped Cone: Apollonius of Perga, Rene Descarte, History of Mathematics etc.

I always thought Co-Ordinate Geometry had something to do with the Conic Sections’ being defined based on the Double-Napped cone. I believed Rene Descarte was the one who made that decision. I thought he did this to keep the definition symmetric about the origin of the [x, y, z] space. That is, I thought, a double-napped cone was conceived so as to give one cone the positive z-coordinates and the other one the negative z-coordinates for symmetrical considerations. Still not clear? Keep reading…

DoubleCone

Figure 1: The Double Cone or the Double-Napped Cone By DemonDeLuxe (Dominique Toussaint) (Own work) [GFDL or CC-BY-SA-3.0], via Wikimedia Commons

I was going through some historical documents related to the Conic Sections and their development when I  came across a name that cleared everything for me:

Mr. Apollonius of Perga (262 BC – c. 190 BC)

He walked on this earth many centuries before Rene Descarte did. He did his Mathematics long before the invention of Co-Ordinate Geometry. Co-Ordinate Geometry basically helped in the Algebraic treatment of Geometry, which was never a concern or a possibility in the times of Mr. Apollonius. He did not use Algebra. He used words and figures. Still, he defined and described the properties of Conic Sections in such methodical and precise way that not much needed to be added ever after. He covered all scenarios. He clearly explained how Conic Sections could be defined with complete accuracy using an oblique cone too:

Cone_3d

Figure 2: Regular Cone on the left and Oblique Cone on the right By DemonDeLuxe (Dominique Toussaint) (Own work) [GFDL or CC-BY-SA-3.0], via Wikimedia Commons

I was surprised beyond limits when I came to know that Mr. Apollonius was the one who defined Conic Sections based on double-napped cone. I thought hard and understood the need for such a definition. A deep look at the following figure will answer all questions:

ConicSections

The Conic Sections By Duk at English language Wikipedia

The co-ordinate system is not explicitly shown in the above diagram. So, let me borrow it from the first diagram. This will keep things simple. Thus, the point where the apex of the two cones meet will be treated as the origin.

Ellipse

When a plane cuts the cone the way it does in the second image of the above diagram, it forms an ellipse. Simple.

Circle

A special case of ellipse where the plane that cuts the cone is parallel to the X-Y plane according the scheme chosen by me.

Hyperbola

The third image in the diagram is a hyperbola. As can be seen, the plane cutting the cone can be at any angle but never equal to the slant angle of the cone. This has a very important implication. The plane cuts both cones of the double-napped cone. The third double-napped cone of Figure 3 shows two hyperbolas. In one the cutting plane is at an angle to the axis of symmetry of the cone while the other plane runs parallel to it. In all the cases, the sections formed on both cones will be symmetrical. The image 3 of the Figure 3 may be misleading but we need to keep in mind that we are dealing with infinite geometrical structures. We, thus, need to extend the image in our minds and see the symmetry.

Parabola

I see this as a very special case of hyperbola when dealing with single cones, but with a double-napped cone this relation is ignored. The plane cutting the cone is at an angle equal to the slant angle of the cone. Thus, it runs parallel to the slant at a distance (if it runs on the slant, we get a line). This also means that the plane can cut only one cone. The other cone will see the plane running parallel to it.

Why is a double-napped cone needed?

I detailed a lot of stuff but never explained why we need a double napped cone if it is not for the satisfaction of co-ordinate system symmetry. Well, a double-napped cone is not needed for the definition of any conic section apart from the hyperbola. The hyperbola is the only section that will be formed on both cones if we have a double-napped cone, but can also be defined very well on the single cone. Also, co-ordinate geometry was not available to Apollonius and there was no pressure on him from Rene Descarte. It seems there is no pressing need for a double-napped cone and a single cone would have sufficed:

ConicSectionsDefinedOnSingleCone

Conic Sections Defined on Single Cone by Magister_Mathematicae

This is what A History of Mathematics by Carl Boyer and Uta Merzbach has to say about double-napped cone in Chapter 7 on Apollonius of Perga:

TheGenesisOfaDoubleNappedConeThus, symmetry was obviously a consideration. Also, the concept of infinity was taken care of. Greeks would never have wanted to face the wrath the Gods by neglecting symmetry and infinity:)

Why do we need a cone for defining ellipse, circle, parabola and hyperbola?

Because we are dealing with conic sections and conic sections can only be defined on a cone. That is a dumb answer, I know.

Try Cylindrical Sections, instead. There are only two curves possible when a plane intersects a cylinder – a Circle and an Ellipse. Nothing else.

What about Spherical Sections. Oh no! Let us not go to that domain. Let us reserve that for some later advanced research. One thing that needs to be understood is that cone and cylinder have single curvature. Sphere is a different beast. It has double curvature. What this means is that a cone or a cylinder can easily be developed:

Development of various Solids from IIT Guwahati Learning Material

Development of various Solids from IIT Guwahati Learning Material

Development of Cone By Cdang (Own work) [CC-BY-SA-3.0], via Wikimedia Commons

Development of Cone
By Cdang (Own work) [CC-BY-SA-3.0], via Wikimedia Commons

A sphere cannot be developed satisfactorily due to the double curvature. If a plane cuts a sphere, the only curve it can generate is a circle. The largest circle that can be formed is when a plane cuts the sphere in two equal halves:

Spherical cap By Pbroks13 (Image:Spherical cap.gif) [Public domain], via Wikimedia Commons

Spherical cap
By Pbroks13 (Image:Spherical cap.gif) [Public domain], via Wikimedia Commons

I will go into the details of space curves (as opposed to plane curves that we have been dealing with in the whole article) cutting spheres and the shapes thus formed later. For now, I have to go deep into the conic sections and their practical applications, analytical geometry etcetera.