Monday, April 23, 2018

Creating a KVM Lab

In the course of my day job, I often have to create VM's to reproduce issues. With my local system this is quite simple, however how a hypervisor is setup to quickly deploy VMS' with commands, like the ones below, require 3 simple primitives to get the VM's to behave the way you expect.

In short you need a Network (I use a NAT network, because I like the isolation), a Storage Pool (I use an LVM backed pool for performance), and DNS (so you don't have to be rain man to access your systems).

Before I get to the Hypervisor setup, you might ask how I create these VM's: 

Creating a VM using a qcow cloud image, and cloud-init
sudo virt-install --name rhel7.5 --vcpus 2 --ram 4096 --disk Downloads/rhel-server-7.5-x86_64.qcow --disk cidata.iso,device=cdrom --disk pool=default,size=15,bus=virtio --network network=default --import --noautoconsole
Creating a VM using and ISO and a KickStart File
sudo virt-install -n rhel7.5 --vcpus 2 -r 4096 --os-type=linux --disk pool=default,size=10,bus=scsi --disk pool=default,size=15,bus=virtio, --network network=default--location=Downloads/rhel-server-7.5-x86_64-dvd.iso --initrd-inject=kickstart.ks -x "ks=file:/kickstart.ks" --noautoconsole
As you can see, with this setup I start from a common install source (base) set of images, that you can pull from a provider (RHEL, CentOS, or Fedora, etc). The second part  that you will find interesting about these commands is the network that I bind the VM's to, and the disks that I create for the VM's to use.

In both examples, and extra 15GB disk is provided to each VM (usually allocated to /dev/vdb, because these are virtio disks). This disk, is created in my default storage pool.

To see what pools you have and/or what resources they provide, you can use the following:
[sudo] virsh pool-list
[sudo] virsh pool-info default
If you need to create a storage pool (an LVM pool, using a blank block device) you can run the following:
[sudo] virsh pool-define-as --name virt-pool --type logical --source-format lvm2 --target /dev/DEVICE
Should you need to, clean up from creating such a pool, the following commands completely clean up from this:
[sudo] virsh pool-destroy virt-pool
[sudo] virsh pool-undefine virt-pool
[sudo] virsh pool-delete virt-pool[sudo] vgremove virt-pool
The network that the VM's are attached too is similarly defined:
[sudo] virsh net-create network_config.xml
File Example:
  <forward mode='{nat'/>
  <bridge name='virtbr-100' stp='on' delay='0' />
  <domain name='minlab.lan' localOnly='yes'/>
  <ip address='' netmask=''>
      <range start='192168.100.1' end='' />
As you can see from this, the VMs I put in this network dhcp boot, using an IP from the defined range(.1 to .253).

Should you need to, clean up from creating such a network, the following commands completely clean up from this:
[sudo] virsh net-destroy minilab
[sudo] virsh net-undefine minilab
[sudo] virsh net-delete minilab
However, even with these tools (the network, and the storage pool) yet another component is needed, to make your lab work as it would in say AWS, GCP, or Azure. This missing component is DNS.

To setup DNS on the hypervisor, the simplest thing to do is have NetworkManager use dnsmasq as a caching server, and point at the Network above's dnsmasq instance to get DNS names for the VM's on the network.

You can enable dnsmask with NetworkManager in 2 ways: 
You can place this in either /etc/NetworkManager/conf.d/localdns.conf or /etc/NetworkManager/NetworkManager.conf and simply restart NetworkManager, to being using this.

However .... you still need to define what DNS servers you want to reference.  This is done by placing a forwarder configuration (as stated above) that points to the dnsmasq instances (per network) started by libvirt.

Example Configuration:
  • Note: This configuration is finky (as in make sure you read the man page, and don't mess it up as debugging can be extremely challenging and painful)
This is placed in /etc/NetworkManager/dnsmasq.d/libvirt_dnsmasq.conf or a similarly named file in the .d directory.

While this process gives you fine grain control over your system and the VM's that you provision, its not overly FAST for creating large lab deployments (more than 2-3 vms), or helpful if you plan to create / delete your labs (as I do).

As a result automation is almost always needed. To simplify the pocess above, helps, by allowing me to define an inventory file (lab) that can be re-used or modified to fit my needs.

Tuesday, April 3, 2018

CA Certificates and python Requests with Fedora

So you want to make an API call with python you say....
You say the API is secured with https (or an TLS/SSL certificate)....

Should be simple right?

No not so much. The short story to this is that Fedora by default wants to use the system certificates stored in /etc/pki/tls/certs/ca-bundle.crt (you can see how this works, by looking at the /usr/lib/python2.7/site-packages/requests/ file.

However if you have the certifi python package being installed (either with pip, or via rpm).
  • (pip) certifi (2018.1.18) - Python package for providing Mozilla's CA Bundle. 
  • (RPM) python2-certifi.noarch 
You CA certificate changes! Changes to /usr/lib/python2.7/site-packages/certifi/cacert.pem

As you can see from: /usr/lib/python2.7/site-packages/requests/ or:$ python -c "from requests import certs; help(certs)"

We on fedora/rhel/centos default to /etc/pki/tls/certs/ca-bundle.crt for our certificates. This also assumes that using this certificate as a CA works.

You can use the following to test (provided you have a cluster_name):
$ curl --cacert /etc/pki/tls/certs/ca-bundle.crt
However with the certifi package installed, it no longer does, so if you go looking for a solution on the inter-webs you might get fun solutions like:
  • r = requests.get(url, verify=cafile)
  • r = requests.get(url, verifiy=False)
  • export REQUESTS_CA_BUNDLE=cafile
All of these can be found as part of

However as a fedora user, I feel the best options are to remove certifi so that it no longer conflicts with the default OS CA path. (should correct the SSL issues):
sudo pip uninstall certifi
sudo [dnf|yum] uninstall python2-certifi
So verify where your request CA is with:
$ python -c "from requests import certs; print certs.where()"
And if its not pointing where it should, it might be that OS is allowing the requests package to use an optional add on to set the path for you, based on what that package provides for the CA.

tl;dr don't rely on an optional dependency, use your hosts CA so that you have consistency between your tools.

Wednesday, December 20, 2017

Fixing Powerline on Fedora (IE: Configuring it Properly)

So I recently got a new work system, and had to re-build my system. As I use powerline to help me visually see whats going on in my bash shell, proved to be VERY helpful.

However one of the things you will not see if you follow this are, git or additional prompts (context queues). To get those you need to set/ configure the proper theme (default_leftonly).

If you read the docs:
You like me will likely be wondering, how do I do what?

So the long and short of how to configure the proper theme, is by running:
# mkdir -p ~/.config/powerline
# cat <<-'EOF' > ~/.config/powerline/config.json
     "ext": {
         "shell": {
             "theme": "default_leftonly"
# exec bash

Monday, January 30, 2017

More on Kubernetes/OpenShift JSON Path

In a past post:

I discussed what json path could do. Today I learned that filters (are only capable of doing simple comparisons (==, !=, <=, >=) type operations.

The tl;dr version is (see regex):

 However because Kubernetes (and by proxy OpenShift) use the jsonpath extension from exponent-io and the golang re2 (regex engine).

The above "filter" operation to find and compare parts of the path are only (as of right now) capable of simple operations.

Thursday, January 26, 2017

Learning / Using jsonpath with OpenShift and Kuberneties

I work as a support engineer on OpenShift and I spend a lot of time trying to get information out of the cluster and often times have to use that to information as an input of some other task I am doing.

Because OpenShift / Kuberneties (cli) have such a wide range of outputs:
-o, --output='': Output format. One of: json|yaml|wide|name|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=... See golang template [] and jsonpath template [].
I have a wide range of options to choose from when trying to complete my tasks. However one is far superior than all others (IMO).
  • Note: Many places on the web reference: as the source for where jsonpath started. 
Why is this better than the others?
  1. Well for starters it lets me get just the output I need. 
  2. It lets me format the data as I see fit. 
  3. It simple to understand and use (compared to go-template)
The best way to show you how easy it is to use is to show you some examples.

Lets start with something simple. Like getting the name of pod.
So if you have the following:
# [oc|kubectl] get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-11-57g56   1/1       Running   0          ---
registry-console-2-49irw   1/1       Running   0          ---
router-1-vinzu             1/1       Running   0          ---
One might just do something like:
# [oc|kubectl] get pods --no-headers | awk '{print $1}'
Simple enough, but if you had to pair this with another command you might start to see issues, with command nesting.

A simpler way (IMO) to get the same information is to do something like:
# [oc|kubectl] get pods -o jsonpath='{.items[*].spec.containers[*].name}{"\n"}'
  • Note: the '{"\n"}' in this command is so that a new line is added at the end of the output, and your terminal does not displaying on the same line as the output.
This allows me just to pull the exact data I want out of the system, and in this case (return it as I want - as a list of name).

If I wanted to do something more complicated, such as get a specific service name and IP from the system, I could easily format the output, and filter (a set of objects) to get me data on a specific object.
# [oc|kubectl] get services -o jsonpath='{.items[?("registry-console")]}: {.items[?("registry-console")].spec.clusterIP}{"\n"}'
Example output:
As a result of output like this, I can get the name of the service, as well as the IP that corresponds to the service. I also get it formatted, exactly how I want it, without having to do complex command stringing (with bash). 

One of the only downsides to using jsonpath is the docs, around the syntax.

If you read, over there is enough, to do damage and shoot yourself in the foot (only to find out your just doing it wrong).

So to help simplify that, I'll look at the same examples above, but with the json they provide.

Whenever starting out its best to just run:
# [oc|kubectl] get <object> -o json > file.json
Simply put, it's simpler to read the json, and then traverse through it to define your jsonpath syntax.

Example JSON  
this is a fedora paste bin an may not longer be around, sorry # [oc|kubectl] get pods -o json > example.json is how to create this)

If you look at my example json, you can see in both examples, that because the CLI gives me information on multiple objects, it creates and returns a List (which if you used to python terminology its a dictionary object, that defines a key, whose name is "items" that's value is an array):
     "kind": "List",
     "apiVersion": "v1",
     "metadata": {},
     "items": [
Because of this, you often have to start all jsonpath filters with:
As your going to be accessing the items array, from this output. However you might not need to do this. Say for example if you got the output of a specific pod.
# [oc|kubectl] get pod registry-console-2-49irw -o jsonpath='{.spec.containers[*].name}'
Because, this command returns a Pod. I don't have to access the member of list (or array).
    "kind": "Pod",
    "apiVersion": "v1",
    "metadata": {
So the first major tip I have with jsonpath is learning how [*] and ['index'] work. ['index'] is exactly what you would expect if you were working with a python dictionary, and when you try and access as key, to get its value. * is a special character, with [] that lets you traverse all the keys (indexes) in the array.

In 90% of what I do, [*] works, as shown above .items is one of the most important thing I have to traverse. So understanding that I am simply looking at the keys, for each member of the array, when specify the next '.' following [*] makes understanding the syntax much simpler.

The next most complicated concepts to learn with jsonpath are filters, and the 'current object' operator. As shown in the example above, ?() is used to create a filter, or a simple 'if TRUE' type of syntax. In the example I used, I used this with in an array [], to identify a pod by its name. The only way this works, is if I can access the 'current object' as I pass through the array. This is what the @ operator does.

With these concepts, your almost cretin to understand jsonpath, and you'll be able to get almost any data out of OpenShift or Kuberneties.

However, one item I did not show an example of, there is another way to traverse arrays. This method uses the range and end keywords.

If we go back to the services example, you can traverse the service and create output (that look like arrays) with something like:
# [oc|kubectl] get services -o jsonpath='{range .items[*]}[{},{.status.capacity}] {end}{"\n"}'
With this you might find it more natural, when traversing the array. However, due to the syntax length (I often don't use this method).

Sunday, January 8, 2017

Fedora and Native Script

I have decided to look into mobile application development. Due to my hardware limitations (I don't own a Mac), I am limited to Android Application development.

However as Java is not my favorite language and I want the ability to possibly port my application over to IOS, I have decided to look into writing an application using NativeScript. This will allow me to learn javascript (better) as well as write an application that has the possibility of being portable to Android and IOS.

To get started, I needed to download and install all of the required dependencies. The NativeScript website has a guide however no instructions for Fedora.

Because I have a newly installed Fedora 25 system, I decided to see if Fedora's DevAssistant could help me get the required dependencies.
  • Note: The Fedora Magazine has a good guide for installing Android Developer Studio. 
    • Installing Android Developer Sudio is a LONG process, as you can dowload up to 50GB+ of material.
    • Your going to need Disk Space for the SDK and the Emulators. 
To get all the needed components, I did the following:
sudo dnf install gcc-c++.x86_64

sudo dnf install devassistant devassistant-ui
da pkg install android-studio
da crt android-studio --name Test

### Setup bashrc
#### edit: ./.bashrc and add the following.

export JAVA_HOME=$(alternatives --list | grep java_sdk_openjdk | awk '{print $3}')

export ANDROID_HOME=${HOME}/android-studio/android-sdk-linux
export PATH=$PATH:${ANDROID_HOME}/tools/
export PATH=$PATH:${ANDROID_HOME}/platform-tools/
### End of file. 

rm -rf Test    ## The project crated by DevAssistant is not needed.

sudo $ANDROID_HOME/tools/android update sdk --filter tools,platform-tools,android-23,build-tools-23.0.3,extra-android-m2repository,extra-google-m2repository,extra-android-support --all --no-ui

sudo npm install -g nativescript --unsafe-perm

tns doctor    ### Make sure it does not report any errors
With this in place I am ready to start the development of my project!

Sunday, October 2, 2016

Using Docker to reposync RHEL content to a systems local storage.

I recently got the opportunity to go on site with a customer, and as such would be on a plane for some period of time (5+hrs). Because of this long section of down time, I wanted to use the time to do some work. However the work I want to focus on requires that I have resources. I plan to work on scripting or automating the deployments of OpenShift with KVM,  but due to my flight (travel accommodations) I will not be connected, or don't want to rely on slow / unreliable internet speeds for package downloads.

Because Satellite (a repository manager) is bulking and my laptops resources are limited.  I decided that hosting the content with a web server an syncing it would be the fastest and most light way to accomplish my goals.

However, as I run Fedora, getting the repositories (packages) was going to require some ingenuity, because reposync only works if you can attach to the repositories, which requires a RHEL server (with a subscription), this means installing a VM, which takes time.

Luckily we have containers! Which means that instead of installing RHEL, in a VM, and mounting a filesystem into the VM. Or turning this VM into the hosting server for the content. I can simply mount my storage (in my case this was an SC card), as part of a RHEL7 container, and use the container and its tools to do what I need to get the content.
$ sudo docker pull
$ sudo docker run -it --rm -v /var/run/<user>/<drive_name>:/opt rhel7 /bin/bash
From here the steps to get the content may vary, but the basic process goes like:
# subscription-manager register --username <RHN_USERNAME> --auto-attach
  • Note: Extra effort, to enable channels might be needed, depending on how auto-attach does, at selecting your subscriptions.
# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
# for repo in rhel-7-server-rpms rhel-7-server-extras-rpms rhel-7-server-ose-3.3-rpm; do reposync --gpgcheck -l --repoid=${repo} --download_path=/opt/ --downloadcomps --download-metadata
# for repo in rhel-7-server-rpms rhel-7-server-extras-rpms rhel-7-server-ose-3.3-rpm; do createrepo -v /opt/${repo} -o /opt/${repo} -g /opt/${repo}/comps.xml; done
Note: I based the repositories (and some of the process) I would need off of -

Because of this method (using Docker), I don't need waste time installing a VM to do these operations, or use unnecessary resources to host this content. This leaves me more room for the VM's I plan to run as part of my scripting / testing, and allowed me to get the content faster than I originally expected.