Announcements on BeyondVM

BeyondVM II: The Blog Awakens

Mon, 07 Dec 2015 18:36:59 -0700

It has been quite some time since I have written anything on here (Almost 1.5 years!) and quite a bit has changed in that time. You probably noticed I re-designed the whole site, but what you may not have noticed is that I ported the whole thing to a static blogging engine called Hugo. This process is something that has been on my to-do list forever and kept me writing anything. The more I wrote the more I would have to convert (Hugo uses Markdown to store its posts, Wordpress does not), so I kept putting it off. More on that later.

I also have shifted the focus of my work towards automation and immutable infrastructure and I have quite a bit to write about the things I have learned doing that I have learned along the way. I am pretty excited about a few of the posts I have brewing on those subjects.

Hugo

Just a short blurb about Hugo. Hugo describes itself as

Hugo is a static HTML and CSS website generator written in Go. It is optimized for speed, easy use and configurability. Hugo takes a directory with content and templates and renders them into a full HTML website.

What this means is that you write posts in markdown (by default) and create templates surrounding the content, and Hugo renders out a fully static website (meaning it renders to plain old HTML). This is a huge advantage because it allows me to host this entire site using services like Amazon S3 fronted by Amazon CloudFront which gives me a really fast, scaleable and cheap hosting solution. I also found a sweet tool called Netlify that I ended up using because they have a bunch of other tools that I don’t have to manually create (minify, etc) but I will include both solutions in the forthcoming article.

I have a blog post in the works about how to build a site using Hugo with some good sane defaults and example code which I found digging around. I also have a set of scripts as well as the template I built for this website that I plan to release shortly (still making some final tweaks).

Focus

In my recent work, I have had I have found myself becoming more and more interested in automation and automation tools as well as the new buzz-word “DevOps.” It is great that someone coined the term DevOps, because it really is a philosophy I have been trying to work under for much of my career - Ops and Dev should really work together as much as possible, no ‘over the fence’ crap - and its great that other people are talking that way now too.

Over the past year or so I have put quite a bit of time into learning some great tools, and I can’t wait to share some posts on them. Here is a short list of some of the topics I have been learning about:

Puppet
Ansible
AWS
HashiCorp Vagrant
HashiCorp Vault
HashiCorp Packer

I am pretty excited about this new direction and I can’t wait to share!

Missing Posts

You may notice that there is some missing content, I only ported the posts that I thought people would still find valuable, so I may have missed something. If you are looking for a particular post and its missing I do have them all archived.

Alert: Firefox 30 is not a supported browser for vCloud 5.1.x

Thu, 19 Jun 2014 00:00:00 +0000

Update (8/4/2014): I have created a tutorial on how to create a ThinApp package to help get around this, check it out!

If you suddenly start seeing something the following error in vCloud Director 5.1.x:

This is because your Firefox upgraded to Firefox 30 automatically, as it does. This error seems to be due to some sort of change that Mozilla added into Firefox 30, I haven’t tracked it down yet (if anyone has let me know!). Even force enabling the plugin won’t help, Firefox 30 looks to have gone to an whitelist only model, doing so will make this error go away but the console sessions never connect.

That being said, VMware has stated that Firefox 30 will not be a supported browser for vCloud 5.1.x (the VMRC plugin doesn’t actually function so it is not a question of only supportability). Unfortunately it is also not supported in IE 10, 11 or Chrome 35 so the only option is really Firefox 29 (or Firefox 3.6). The only solution is to upgrade to vCloud 5.5.x, which while true isn’t really a solution.

As a note, I have had success using ThinApp to isolate a Firefox 29.0.1 with the VMRC plugin, Flash and Java and Portable Firefox may be an option too. Either way, be sure to disable auto-update (Firefox Options -> Advanced -> Update) if you downgrade to Firefox 29.0.1. There are some tips here on how to do that, the big one being the ‘-no-remote’ option to allow the ThinApp Firefox to launch its own process.

Update: It seems that installing the Firebug plugin and enabling that will allow the console to connect in Firefox 30 after forcing the plugins to activate. While cumbersome, this is another workaround.

Update 2: (7/29/14): I have found another solution! It is possible to use the Portable Firefox ESR Version (Extended Support Release), this is a Firefox 24 package that will run in a sandbox. It will use the installed plugins on your system so if your Java version is also incompatible that won’t work. One thing to change, perform the following:

Locate FirefoxPortable.ini in your extracted ESR download (might be in Other/Source directory).
Copy to the root of Portable Firefox ESR install directory
Edit file and change the “AllowMultipleInstances” variable to true.
Save

This will allow the Portable Firefox and Installed Firefox to run at the same time.

Alert: Java JRE 7u51 breaks Everything

Mon, 27 Jan 2014 00:00:00 +0000

Update (8/4/2014): I have created a tutorial on how to create a ThinApp package to help get around this, check it out!

This morning it came to my attention that my customers were no longer able to upload any media (OVFs or ISOs) to their vCloud catalogs. This seems to be due to the most recent Java JRE version released by Oracle. The behavior I experienced was that the applet would appear to load but when I would click on the browse button nothing would happen. This happens across all different browsers and browser versions. The reason for this seems to be a change in the requirements for certificates and applet signing in JRE 7uU51.

Luckily, there is a quick workaround for this:

Open Configure Java control widget, navigate to Security then Edit Site List.

Type the URL for your vCloud instance in the blank and Click Add to add (repeat for multiple URLs), click OK

Thats it! This really is a workaround, I am not sure what the final solution will be but this got my customers and I back into business for now. If anyone has more information, let me know!

Update: It seems that this update actually breaks everything from HP iLO to vCenter Orchestrator, I am still searching for a better workaround if anyone knows of one.

Update 2: There are reports of this breaking Cisco UCS, Dell iDRAC, vCloud VPN and EMC Unisphere as well. It might be safe to say that most Java based management tools are going to be effected. VMware has released a KB article about their parts of it, check it out here. One interesting thing in the KB is that the upload portion of this issue does not effect vCloud 5.5 as this uses the Client Integration Plugin.

Update 3: Setting the security level to “Medium” also seems to work for most things.

vOpenData Update – Dashboard v2.0

Wed, 08 May 2013 00:00:00 +0000

A few days ago I pushed live a new update for the vOpenData Dashboard which included a few interesting things from an application perspective, I wanted to highlight some of them here.

###The Code

The first major thing is that the first version of the dashboard was written using a framework called Dashing which is a sweet frame work for developing dashboards from any datasource really quickly but, as it turns out, is not great at efficiently handling hundreds of connections.

Because of this I re-wrote the dashboard from scratch using a simple Sinatra app, it ended up only being about 40 lines of actual logic (not including the HTML part) to get the job done, pretty awesome.

This simplicity also allowed me to throw some memcache caching up infront of it to handle just about any traffic I can throw at it. I actually tested up to 250 hits/second using blitz.io and it functioned flawlessly. Awesome.

I also used a totally sweet framework called isotope to do the layout of the tiles on that page that will allow me to do some even cooler things in the future. One thing I did add though was the ability to filter by category, just click on the buttons in the toolbar to drill down a little, you can get views like this for LUNs:

###The Stats

In this version we wanted to bring to the front a few new stats, most importantly being the storage vendor statistics.

Storage Stats – These stats may be interesting to some folks as NetApp is not among the top storage vendors on this list. My theory on this is because NetApp admins typically (from my experience) provision large NFS volumes versus smaller VMFS volumes. This stat is based on the number of LUNs not the total storage provided by each vendor (which would also be interesting to see, that will come later) so this list did not really surprise me personally. As a little tease for what is to come, I included a screenshot of the top 15 vendors sorted by amount of storage, interesting results as well.
Environment Types – Another thing we decided to add was a breakdown of the environment types that we have gotten so far. We thought this would be a great thing to display as it would give everyone an idea of what kind of data we have so far and who we need to bug as a community to upload. I was happy to see that we had a huge portion of the environments being from production server based infrastructures, awesome work.
Host Hardware – We increased the number of host vendors to 5 (after many asks from all of you!) and this shows basically what I expected as well. I did expect Cisco to have a higher percentage but I suppose those are the larger environments that may not have reported in yet.
Country Standings – One note about these, I was actually not calculating these correctly (dividing submissions by number of users, doh!) so that is why these seem to have changed drastically. The data is the same, I just am mathing correctly now.

###Data Curation

People were asking if we combine DGC (EMC Clariion based storage) and EMC?

It seems like the census is to spend some time curating the data, combining Dell and Dell Inc for example, so I will be working on some code to do that as part of the next update. This will probably be asynchronous (run as a job on the whole dataset as a whole on a schedule) so it won’t necessarily do it instantly on upload, this is to keep the time it takes to insert the data to a minimum.

###The Future

Next up, the future. I have been doing a lot of thinking about where to take this project in the future. I was really astounded by the response that we have gotten from the community, you all have blown me away. I really have some cool ideas for this project that I am not quite ready to talk about but I think everyone will really love.

My goal is to be able to give everyone the ability to mix this data as they want to see what the community can come up with with all of this data, I am sure some of you have some amazing ideas. If you do have anything in particular send me an email and I would love to chat about it.

I also am looking to give some filtering and mashing up of the data on the private dashboards to narrow the scope of some of the averages, I know that is what people are really looking for.

One more thing (channeling The Steve), the Microsoft sized gorilla in the room. I have thought about the ability to gather stats from other vendors environment in a similar manor to compare and I think this would be really amazing. If any one is (or knows of) a rockstar that can work with the Hyper-V, Citrix, or OpenStack APIs get them in touch with me on twitter or by email.

Until then my head is going to be buried in refactoring and improving the main site codebase to get something really great out to the community, stand by.

vOpenData – Crunching Everyone’s Data For Fun And Knowledge

Fri, 12 Apr 2013 00:00:00 +0000

It has been quite some time since I have gotten a chance to write on this blog but recently I was able to work with Willian Lam (@lamw) on a really awesome project so I thought this was a great opportunity to start up again. This project really started from a tweet by Duncan Epping (@duncanyb):

This really got me thinking, I get asked these questions all the time and there really isn’t a good answer to it. Since I am going to need to answer this question more often in the future I decided that I would take on this challenge, thus vOpenData was born.

First a little about vOpenData:

###What is vOpenData?

vOpenData is an open community project that grew from the question “What is the average VMDK size for deployed virtual machines?” We wanted to create an open community database that is purely driven by users submitting their virtual infrastructure configurations. Leveraging the powerful virtualization community and applying simple analytics we are able to provide various trending statistics and data for virtualized environments.

###What information do we collect?

We made an effort to not collect specific information such as hostnames or even display names that could be used to identify a particular organization. Instead, we are using UUIDs which are automatically generated by the virtualization platform to uniquely identify a particular object. This allows us to keep track of changes in the our database when a new data set is uploaded from an existing environment. In addition we are collecting various configuration data and you can find a complete list in the Data FAQs

To see more detail on the project and how to get involved, see the Virtually Ghetto article here: VirtuallyGhetto

I am really excited to see what data everyone is able to provide. The one thing that you will notice is that not all of the statistics are available on the public dashboard; this is to try and get you to get involved by uploading data. There will be a myriad of statistics and reports that we plan on making available to people that upload data into the system.

Now for some gory details, that is why we are here, right? So, to solve this problem in any type of speedy manor I needed to use a tool that was designed for rapid prototyping and development. I recently finished another project using Ruby On Rails so I elected to use that. Going forward I will assume that you have a basic understanding of Ruby On Rails, if not read about it here.

###The Code

With Ruby On Rails I was able to develop the entire application with very little actual code written. In fact there is a built in tool for just this; here are the stats:

From this you can see that I only had to write about 1000 lines of code for this project, which is really awesome. This doesn’t include the rake tasks that actually calculate the stats, these were another 400 lines or so but are nothing overly complicated.

Another tool that saved me a ton of time was using the Twitter Bootstrap layout framework. (More here) This allowed me to quickly build out a usable and good looking interface in no time, if you have never used it I totally recommend it. Also, there are a few sweet add-ons that allow quick theming of applications; this is a great collection.

Another awesome feature of rails comes from ruby and that is the plug-ability of the gem system. Because of this I was able to use a few select libraries to stand on the shoulders of giants and get this project rolling quickly. My basic stack included the following gems:

I recommend checking all of those projects out, there are some really smart people behind all of them.

###The Platform

The next issue I had to solve was that of hosting all of this data. Due to the fact that I am a lazy sysadmin and have little free time I elected to use a PlatformAsAService (PaaS) for this project so I didn’t have to spend any time building servers, the one I chose was Heroku.

I picked heroku for the following reasons:

Its fast.
It’s quickly scalable, I can add worker and background processes on demand with little effort.
It’s pluggable, I can easily integrate it with many services like RedisToGo (for Resque) and Postgres.
My default avatar was a Ninja (Ok, not really)

The quick and dirty is that heroku allows you to build a rails app locally and push it (using git) to their production stack with no effort. This allowed William and I to rapidly update the code and discuss changes in near real time. The detailed explanation is lengthy and warrants its own article, more on that later.

Once I chose heroku I needed a place to store all of these uploaded CSV files until I could get them processed (a lengthy process which is done in the background). For this I chose the venerable amazon S3, there was a pretty good plugin (Paperclip) that easily handles the amazon process and allows easy local caching of the files on the workers while they were running. The basic data flow is this:

You upload data to vOpenData
The data is copied to S3 for temporary storage
When your infrastructure’s turn comes up in the queue the data is cached on the worker instance
The worker inserts the data into the database

Basically for this project S3 is just acting as scratch space until the data can be chunked into heroku.

The last piece of this project was the public dashboard. For that, William came up with an awesome tool by Shopify called Dashing. Dashing is a basic rack app that runs scheduled jobs to fetch data from an API and push them to a live dashboard.

That is basically it, if there are any specific questions leave them in the comments and I will answer them, I am also interested in writing a few follow-up articles if the demand is there to explain some of the code in depth.

William and I submitted a VMworld session to talk about the data that we have collected so far and get feedback from the community, it is “4976 – vOpenData – Crunching Everyone’s Data For Fun And Knowledge” so be sure to vote for it!

One last thing, we plan on adding lots of incremental changes over the next few months to add more features (like more dashboards and data rollups!) so be sure to follow myself (@wazoo), William (@lamw) and the vOpenData (@vopendata) user on twitter to get updates!