Monday, January 30, 2017

More on Kubernetes/OpenShift JSON Path

In a past post: http://sferich888.blogspot.com/2017/01/learning-using-jsonpath-with-openshift.html

I discussed what json path could do. Today I learned that filters (are only capable of doing simple comparisons (==, !=, <=, >=) type operations.

The tl;dr version is (see regex):

https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/util/jsonpath/parser.go#L338

 However because Kubernetes (and by proxy OpenShift) use the jsonpath extension from exponent-io and the golang re2 (regex engine).

The above "filter" operation to find and compare parts of the path are only (as of right now) capable of simple operations.





Thursday, January 26, 2017

Learning / Using jsonpath with OpenShift and Kuberneties

I work as a support engineer on OpenShift and I spend a lot of time trying to get information out of the cluster and often times have to use that to information as an input of some other task I am doing.

Because OpenShift / Kuberneties (cli) have such a wide range of outputs:
-o, --output='': Output format. One of: json|yaml|wide|name|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=... See golang template [http://golang.org/pkg/text/template/#pkg-overview] and jsonpath template [http://releases.k8s.io/release-1.3/docs/user-guide/jsonpath.md].
I have a wide range of options to choose from when trying to complete my tasks. However one is far superior than all others (IMO).
jsonpath
  • Note: Many places on the web reference: http://goessner.net/articles/JsonPath/ as the source for where jsonpath started. 
Why is this better than the others?
  1. Well for starters it lets me get just the output I need. 
  2. It lets me format the data as I see fit. 
  3. It simple to understand and use (compared to go-template)
The best way to show you how easy it is to use is to show you some examples.

Lets start with something simple. Like getting the name of pod.
So if you have the following:
# [oc|kubectl] get pods
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-11-57g56   1/1       Running   0          ---
registry-console-2-49irw   1/1       Running   0          ---
router-1-vinzu             1/1       Running   0          ---
One might just do something like:
# [oc|kubectl] get pods --no-headers | awk '{print $1}'
Simple enough, but if you had to pair this with another command you might start to see issues, with command nesting.

A simpler way (IMO) to get the same information is to do something like:
# [oc|kubectl] get pods -o jsonpath='{.items[*].spec.containers[*].name}{"\n"}'
  • Note: the '{"\n"}' in this command is so that a new line is added at the end of the output, and your terminal does not displaying on the same line as the output.
This allows me just to pull the exact data I want out of the system, and in this case (return it as I want - as a list of name).

If I wanted to do something more complicated, such as get a specific service name and IP from the system, I could easily format the output, and filter (a set of objects) to get me data on a specific object.
# [oc|kubectl] get services -o jsonpath='{.items[?(@.metadata.name=="registry-console")].metadata.name}: {.items[?(@.metadata.name=="registry-console")].spec.clusterIP}{"\n"}'
Example output:
registry-console: 172.30.12.188
As a result of output like this, I can get the name of the service, as well as the IP that corresponds to the service. I also get it formatted, exactly how I want it, without having to do complex command stringing (with bash). 

One of the only downsides to using jsonpath is the docs, around the syntax.

If you read, over https://kubernetes.io/docs/user-guide/jsonpath/ there is enough, to do damage and shoot yourself in the foot (only to find out your just doing it wrong).

So to help simplify that, I'll look at the same examples above, but with the json they provide.

Whenever starting out its best to just run:
# [oc|kubectl] get <object> -o json > file.json
Simply put, it's simpler to read the json, and then traverse through it to define your jsonpath syntax.

Example JSON  
this is a fedora paste bin an may not longer be around, sorry # [oc|kubectl] get pods -o json > example.json is how to create this)

If you look at my example json, you can see in both examples, that because the CLI gives me information on multiple objects, it creates and returns a List (which if you used to python terminology its a dictionary object, that defines a key, whose name is "items" that's value is an array):
{
     "kind": "List",
     "apiVersion": "v1",
     "metadata": {},
     "items": [
         {
     ...
Because of this, you often have to start all jsonpath filters with:
'{.items[...
As your going to be accessing the items array, from this output. However you might not need to do this. Say for example if you got the output of a specific pod.
# [oc|kubectl] get pod registry-console-2-49irw -o jsonpath='{.spec.containers[*].name}'
Because, this command returns a Pod. I don't have to access the member of list (or array).
{
    "kind": "Pod",
    "apiVersion": "v1",
    "metadata": {
...
So the first major tip I have with jsonpath is learning how [*] and ['index'] work. ['index'] is exactly what you would expect if you were working with a python dictionary, and when you try and access as key, to get its value. * is a special character, with [] that lets you traverse all the keys (indexes) in the array.

In 90% of what I do, [*] works, as shown above .items is one of the most important thing I have to traverse. So understanding that I am simply looking at the keys, for each member of the array, when specify the next '.' following [*] makes understanding the syntax much simpler.

The next most complicated concepts to learn with jsonpath are filters, and the 'current object' operator. As shown in the example above, ?() is used to create a filter, or a simple 'if TRUE' type of syntax. In the example I used, I used this with in an array [], to identify a pod by its name. The only way this works, is if I can access the 'current object' as I pass through the array. This is what the @ operator does.

With these concepts, your almost cretin to understand jsonpath, and you'll be able to get almost any data out of OpenShift or Kuberneties.

However, one item I did not show an example of, there is another way to traverse arrays. This method uses the range and end keywords.

If we go back to the services example, you can traverse the service and create output (that look like arrays) with something like:
# [oc|kubectl] get services -o jsonpath='{range .items[*]}[{.metadata.name},{.status.capacity}] {end}{"\n"}'
With this you might find it more natural, when traversing the array. However, due to the syntax length (I often don't use this method).

Sunday, January 8, 2017

Fedora and Native Script

I have decided to look into mobile application development. Due to my hardware limitations (I don't own a Mac), I am limited to Android Application development.

However as Java is not my favorite language and I want the ability to possibly port my application over to IOS, I have decided to look into writing an application using NativeScript. This will allow me to learn javascript (better) as well as write an application that has the possibility of being portable to Android and IOS.

To get started, I needed to download and install all of the required dependencies. The NativeScript website has a guide however no instructions for Fedora.

Because I have a newly installed Fedora 25 system, I decided to see if Fedora's DevAssistant could help me get the required dependencies.
  • Note: The Fedora Magazine has a good guide for installing Android Developer Studio. 
    • Installing Android Developer Sudio is a LONG process, as you can dowload up to 50GB+ of material.
    • Your going to need Disk Space for the SDK and the Emulators. 
To get all the needed components, I did the following:
sudo dnf install gcc-c++.x86_64

sudo dnf install devassistant devassistant-ui
da pkg install android-studio
da crt android-studio --name Test

### Setup bashrc
#### edit: ./.bashrc and add the following.

export JAVA_HOME=$(alternatives --list | grep java_sdk_openjdk | awk '{print $3}')

export ANDROID_HOME=${HOME}/android-studio/android-sdk-linux
export PATH=$PATH:${ANDROID_HOME}/tools/
export PATH=$PATH:${ANDROID_HOME}/platform-tools/
### End of file. 

rm -rf Test    ## The project crated by DevAssistant is not needed.

sudo $ANDROID_HOME/tools/android update sdk --filter tools,platform-tools,android-23,build-tools-23.0.3,extra-android-m2repository,extra-google-m2repository,extra-android-support --all --no-ui

sudo npm install -g nativescript --unsafe-perm

tns doctor    ### Make sure it does not report any errors
With this in place I am ready to start the development of my project!

Sunday, October 2, 2016

Using Docker to reposync RHEL content to a systems local storage.

I recently got the opportunity to go on site with a customer, and as such would be on a plane for some period of time (5+hrs). Because of this long section of down time, I wanted to use the time to do some work. However the work I want to focus on requires that I have resources. I plan to work on scripting or automating the deployments of OpenShift with KVM,  but due to my flight (travel accommodations) I will not be connected, or don't want to rely on slow / unreliable internet speeds for package downloads.

Because Satellite (a repository manager) is bulking and my laptops resources are limited.  I decided that hosting the content with a web server an syncing it would be the fastest and most light way to accomplish my goals.

However, as I run Fedora, getting the repositories (packages) was going to require some ingenuity, because reposync only works if you can attach to the repositories, which requires a RHEL server (with a subscription), this means installing a VM, which takes time.

Luckily we have containers! Which means that instead of installing RHEL, in a VM, and mounting a filesystem into the VM. Or turning this VM into the hosting server for the content. I can simply mount my storage (in my case this was an SC card), as part of a RHEL7 container, and use the container and its tools to do what I need to get the content.
$ sudo docker pull registry.access.redhat.com/rhel7
$ sudo docker run -it --rm -v /var/run/<user>/<drive_name>:/opt rhel7 /bin/bash
From here the steps to get the content may vary, but the basic process goes like:
# subscription-manager register --username <RHN_USERNAME> --auto-attach
[Password]: 
  • Note: Extra effort, to enable channels might be needed, depending on how auto-attach does, at selecting your subscriptions.
# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
# for repo in rhel-7-server-rpms rhel-7-server-extras-rpms rhel-7-server-ose-3.3-rpm; do reposync --gpgcheck -l --repoid=${repo} --download_path=/opt/ --downloadcomps --download-metadata
# for repo in rhel-7-server-rpms rhel-7-server-extras-rpms rhel-7-server-ose-3.3-rpm; do createrepo -v /opt/${repo} -o /opt/${repo} -g /opt/${repo}/comps.xml; done
Note: I based the repositories (and some of the process) I would need off of - https://docs.openshift.com/container-platform/3.3/install_config/install/disconnected_install.html#disconnected-syncing-repos

Because of this method (using Docker), I don't need waste time installing a VM to do these operations, or use unnecessary resources to host this content. This leaves me more room for the VM's I plan to run as part of my scripting / testing, and allowed me to get the content faster than I originally expected. 

Tuesday, September 6, 2016

Improving the user experience of hub

I can't remember things well,  So when possible rely tools (like bash completion) to help me complete or explore commands with linux so that I don't have to remember every commands 100+ options.

With hub, the same applies and you may not have bash-completion for this new tool. If you install it directly from the hub site. 

To help you solve that, you can do:
if [[ -f /usr/share/bash-completion/completions/hub ]]; then echo "hub bash-completion already installed"; else  sudo curl -o /usr/share/bash-completion/completions/hub https://raw.githubusercontent.com/github/hub/master/etc/hub.bash_completion.sh; fi
This relies on the bash-completion construct but its the simplest way to add completion commands to this new tool.
  • This also updates your "completion" commands for git so be aware that if you have not aliased "git > hub" your completion commands may not correctly display your available options.
It should also be noted that if your not installing the hub tooling from the project site directly (but instead using your OS's package manger), bash completion may already come with the software. Be sure to explore this (and reload your shell after installing). 

Simplifying your github workflow with "hub"

As a support engineering / software maintenance engineer, I spend a lot of time looking at code, reading code, searching code, etc. So needless to say I have a few git repos's sitting arround. 

In most cases, I use these repositories to help me understand errors, or issues that customers see when using a product or library, that I am supporting. However in some situations, the problems once identified are simple to fix, it becomes time for me to change hats and become an contributor.

In most cases contributing its a pain because it involves understanding parts of the SCM (git) tooking that can be difficult to understand (when your first getting started). Depending on the project, contributing, or the workflow in how you provide contributions to the project can be "challenging". Luckily services like github have arisen to make the sharing / hosting of OpenSource projects simple. They have also worked to make "contributing" to projects simple (however these efforts of go with out praise).

One such example is github's invention of the "pull request". While this is a source control concept for "git", however its workflow definition has fundamentally altered, how contributions to projects work because it defined tooling to unify and simplify the process of contributions.


One of the complications with the "pull request" is that it, with out "sevice tooling" (Github) you are more or less providing "patches" that have to be manualy managed. The biggest complication caues by this "sevice tooling" (Github) is that you have to use the "web ui" to create / submitt a PR.

Not any more. With "hub" you can remove this complication, and move back to the terminal.

To get started you will need to install the "hub tooling", in my case on fedora I can just run.
sudo yum install hub
I can then use the use hub like 'git'.
hub clone https://github.com/openshift/openshift-docs
cd openshift-docs/
Except now, I have new options that integrate directly with the github service.
hub fork
git remote -v
From here on out, most of the "contribution process" is likely the same as what you do in for any git project. Branch -> Modify - > Commit - > Push.
git checkout -b typos
< make changes > 
git add install_config/install/prerequisites.adoc install_config/install/disconnected_install.adoc
 

<confirm changes> 
git diff master
git status
git commit -m "fixing typos"

git push <GITHUB_USER> typos
With you changes now in your "fork" and "feature branch" you can switch back to hub to complete your PR and contribute to the project.
git pull-request
  •  Note: you need to ensure that you setup your github ssh key, or you set on your git configuration:

    git config --global hub.protocol https 

Friday, June 17, 2016

Implementing Dynamic DNS in OpenShift v3 (with Python and Free IPA)

For a few days, now I have been trying to think of a way that I could, remove the wild card DNS requirement from OpenShift v3.

In the v2 we had the capability / functionality dynamically update DNS servers, when applications were created. So naturally I went looking for a way to also do this.

It turns out it is possible to monitor the routes API in OpenShift v3 and with the information provided, by the events stream, then update a Dynamic DNS service like IPA.