Introducing Cloudformer, gem for AWS Cloudformation

Cloudformer attempts to simplify AWS Cloudformation stack creation process in ruby projects by providing reusable rake tasks to perform common operations such as apply(create/update), delete, recreate on stack along with validations on templates. Task executions which enforce a stack change will wait until ROLLBACK/COMPLETE or DELETE is signalled on the stack (useful in continuous deployment environments to wait until deployment is successful). Refer github page for more information.

The list of rake tasks provided are:


rake apply           # Apply Stack with Cloudformation script and parameters(And wait till complete - supports updates)
rake delete          # Delete stack from CloudFormation(And wait till stack is complete)
rake events          # Get the recent events from the stack
rake outputs         # Get the outputs of stack
rake recreate        # Recreate stack(runs delete & apply)
rake status          # Get the deployed app status

Installation

Add this line to your application’s Gemfile:

gem 'cloudformer'

And then execute:

$ bundle

Or install it yourself as:

$ gem install cloudformer

AWS Environment configuration

Cloudformer depends on the aws-sdk gem to query AWS API. You will need to export AWS configuration to environment variables to your .bashrc/.bash_profile or your build server:

export AWS_ACCESS_KEY=your access key
export AWS_REGION=ap-southeast-2
export AWS_SECRET_ACCESS_KEY=your secret access key

Configuration

You can add cloudformer tasks to your project by adding the following to your rake file:

require 'cloudformer'
Cloudformer::Tasks.new("earmark") do |t|
  t.template = "cloudformation/cloudformation.json"
  t.parameters = parameters
end

where cloudformation/cloudformation.json is the stack json file and parameters is a hash of parameters used in the template. For a template which takes the following parameters:

"Parameters": {
  "PackageUrl": {
    "Type": "String"
  },
  "PackageVersion": {
    "Type": "String"
  }
}

the parameter hash(Ruby Object) would look like:

{
  "PackageUrl" => "http://localhost/app.rpm",
  "PackageVersion" => "123"  
}  

If you have a template with no parameters, pass an empty hash {} instead.

You can find the open source code in github, any feedback/ pull requests would be awesome :)

Continuous deployment with cloud formation and cfn-hup

Cloud formation lets you create application stacks from scratch in EC2. In my project I use cloud formation to do continuous deploys using cfn-hup with every git push. If you haven’t used cloud formation before, there are plenty of tutorials on the web to get you started.

Structure:

The application is bundled into a RPM using FPM with all the application dependencies and post install scripts to be executed when the RPM is installed.

Packaging and CFN-Script:

As a part of the CI build, RPM of the app is created and uploaded to S3 tagged as ‘latest’ and the cfn-script downloads the RPM and installs it. An example looks like:

curl ‘package_url’ > /tmp/earmark-app.rpm
yum erase -y earmark-app
yum install /tmp/earmark-app.rpm
</code>
CFN-HUP Configuration:
<code>
[main]
stack=stack-name
region=ap-southeast-2

Waiting on Resource Condition:

You can use ApplicationWaitHandle & ApplicationWaitCondition to signal cloud formation deploy to wait until the installation of the package to complete. You can then use cfn-signal to indicate cloud formation stack that the particular action is complete. If cfn-signal isn’t received within a specified time period, the stack creation will fail. Here is an example WaitCondition:

“ApplicationWaitCondition” : {
    “Type” : “AWS::CloudFormation::WaitCondition”,
    “Properties” : {
    “Count” : “1”,
    “Handle” : {“Ref” : “ApplicationWaitHandle”},
    “Timeout” : “300”
    }
},
“ApplicationWaitHandle” : {
    “Type” : “AWS::CloudFormation::WaitConditionHandle”
}

You can then wait on this condition in your stack by using:

“LoadBalancer”: {
    “DependsOn”: “ApplicationWaitCondition”,
}

Signalling Cloud Formation:

You can then signal cloud formation that the install was successful using:

"/opt/aws/bin/cfn-signal -e 0 -r "Application Installation Complete." ‘", { "Ref" : "ApplicationWaitHandle" }, "’\n"

Limitations:

  • Cfn-hup checks for updates on the stack every 15 minutes by default and there is no way to alter the duration.
  • Cfn-hup cannot be used to signal stack wait during stack updates.
  • Cfn-hup failures aren’t propagated back to the stack events which could result in a false positivies even when update fails.

Futurespective: Influencing your team’s future

"Our life is what our thoughts make it." (Marcus Aurelius)

Any team practising(or claiming to do agile) is well aware of the positive outcomes that retrospectives bring into the project. Retrospectives helps us to refocus our priorities based on our past experiences, identifying potential areas of improvement to help us be more productive and increase delivered value. Retrospectives are derived from the parallel in the medical universe - “retrospective cohort study” where individual circumstances are assessed based on a particular outcome(in software engineering world: successful delivery of a project). Like any other representative study, it suffers from cognitive biases(selection and information bias). In software engineering, such biases results in the discussion being skewed towards perceived problems in the past few weeks(assuming retros happen that often). Typically, an action from a retrospective would be geared towards fixing a problem based on its occurrence in the past(you were trying to get more productive after all), which need not necessarily help the team achieve its goal over the next 6 months(or 2 years, depending on how far you look).

To quote Uncle Bob here, “Agile is not just processes and practices but a belief in achievement”. If agile is all about believing in achievement, we should be able to set goals as a team that we would like to achieve in the future and agree that  reality is not necessarily all about harshness and things going wrong all the time. So, once in a while(either during the kick-off of a project or even in the middle of a project), it’s a good exercise to step forward and see where we would like to see ourselves as a team in the next 6 months(or 2 years) and what we would like to achieve in that time-frame. It helps the team to refocus team values and come to a agreement on a common goal and vision, helping us all to be more collaborative in how we can reach the ultimate end goal.

Enter futurespectives. A really retro, retrospective. 

"Imagine yourselves in a kick-ass time machine and you decide to land in the future(6 months) and have a peek how the team is performing. You are intrigued to find out that the team has achieved all it’s set goals and objectives of the project and team member satisfaction is at it’s all time high. How did it happen? What were the things that the team did to reach nirvana?"

Given that there is no actual time machine(jeez spoilers) available, the only option you have is to walk backwards to find out how this parallel universe came into existence. Here are some of key things to think about: 

  1. What’s the *common goal* everyone was trying to move forward to?
  2. What did we do *more*?
  3. What did we do *less*?
  4. What things were *we not sure* that we monitored.

Such a discussion into the perceived future makes few key assumptions:

  1. We all agreed and understood that the team had common goals to achieve.
  2. We were fully aware of the end goals of the project & team and were in the same page.

Though we cannot be really sure such a universe could exist, we all can agree which actions(success factors) will land us on the happy universe and what(failure factors) will land us on the not so happy parallel universe. Using the outcomes of this brainstorm, we could focus our energies to get our act together towards the happy universe. 

Barebones XMonad config for Ubuntu

After countless hours of dabbling with Xmonad, a few really weird bugs, I got Xmonad to work nicely in Ubuntu 12.04 LTS. If you haven’t used/heard of Xmonad before you should check out Xmonad. It’s a simple and beautiful tiling manager written in Haskell and is configured in Haskell. I hope this post helps someone who gave it a go but backed of due to some of the weird issues relating to window focus in Java apps.

Why

  • Kickass multi-monitor support (xinerama) with per screen workspaces.
  • Simple, minimalistic and fast.
  • Kickass WM layouts that you can configure using Haskell.
  • Amazing keyboard shortcuts to re-align windows and navigate seamlessly.

Known Issues

  • Xmonad has some window focussing issues when it comes to Java apps (hack is available which does not work on JDK7)
  • Doesn’t work well with Unity 3d, so you might have to switch to Unity 2d. Alternatively if you are using classic gnome-shell, it should work well.

How to get it

  • If you are ubuntu you can use the traditional package manager aka apt-get to install.

    sudo apt-get install xmonad
  • You should also get some application launcher(Unity dash works, but there are some performance issues). I use dmenu, it’s simplistic and works (smile)

    sudo apt-get install dmenu

Gnome Session Config

Default install of Xmonad in Ubuntu 12.04 LTS adds the necessary session for Xmonad to work with Gnome with the exception of Default-window manager and DefaultProvider-panel. You can add them by editing the /usr/share/gnome-session/sessions/xmonad.session. You add them to session config which should look something like this.

[GNOME Session]
Name=Xmonad Unity-2D Desktop
RequiredComponents=gnome-settings-daemon;
RequiredProviders=windowmanager;panel;
DefaultProvider-windowmanager=xmonad; #this would have been missing
DefaultProvider-panel=unity-2d-panel; #this would have been missing

Issue with Intellij IDEA:

There is a known issue with XMonad not being able to interact with the GUI components when running under JAVA 7, so simplest would be to run under JAVA 6 and use ICCCMFocus#setWMName “LG3D” to ensure Idea to gain focus and play nicely with Xmonad(It’s included as part of the minimalistic xmonad config.)

Minimalistic Xmonad config

import XMonad
import XMonad.Util.Run
import XMonad.Util.EZConfig (additionalKeys)
import XMonad.Hooks.ManageDocks
import XMonad.Hooks.ICCCMFocus
import XMonad.Config.Gnome
import XMonad.Hooks.SetWMName
import XMonad.Actions.DynamicWorkspaces
myManageHook = composeAll (
  [ className =? "Unity-2d-panel" --> doIgnore
  , className =? "Unity-2d-launcher" --> doFloat
  , className =? "gnome-do" --> doFloat
  ])
main =
  xmonad $ gnomeConfig { modMask = mod4Mask
               , startupHook = setWMName "LG3D"
                       , manageHook = manageDocks <+> myManageHook <+> manageHook gnomeConfig
                       , layoutHook = avoidStruts $ layoutHook defaultConfig
                       , logHook = takeTopFocus
                       } `additionalKeys` [ ((mod4Mask, xK_d), spawn "dmenu_run -b")
                                          ]

XMonad Keymap cheat sheet

A vim style cheat sheet for Xmonad is available here.

Using OSX ? Brian McKenna (ex - Atlassian) seems to have patched Xmonad to work nicely under OSX.

Review/Opinion- 2016:Obama’s America

It’s confirmed, we have proof, America will transform itself into a communist country by 2016 and Obama will serve as its most fond dictator - or so is the message of the documentary “Obama’s America 2016”. D Souza and his team attempts to walk through Obama’s past and show us how it influences his actions and policies.  First forty minutes of the documentary explores young Obama’s life, influence of his absentee father and how it shaped his thoughts (base on his  own book - Dreams from my Father). 

He follows his life back to Kenya, Indonesia, about his father’s accidental death and about his mentor who supposedly was a pro-communist. He also points out that Obama Sr. wrote a pro-anticolonial paper and suggested increased taxes upto 100%. So? D Souza who pointed out the similarities with his childhood in India, however D Souza was able to change his opinion on pro-socialistic thought processes that he grew up with, but Obama was influenced by it, even though he hardly lived with his father. He also claims that Obama was pro-seperatist when he grew up in Hawaii, with practically no-evidence whatsoever. All this, is quite a logic stretch.

Apart from these unsubstantiated personal attacks, there are so many factual inaccuracies and careful omissions to construct the innuendo. D Souza suggests that Obama sides with radical Jihadists, that he did not help a friendly regime(Hosni Mubarak, the dictator) wants to reduce military spending(which is already 5 times as much as the next country China), reduce presence of military in Iraq and Afghanistan and hand over the power to rest of the world. Hold on, that pretty much sounds ridiculous. What do conservatives want? Wage war all the time and leave the world in a state of perpetual chaos?

Apart from policies in middle east, in the debate of national debt, he claims that Obama increased the national debt significantly, with the inner lining like, “Hey, America, Obama is trying to take all your money and give it to the poor”. Everybody who does their own fact checking and does not get all the news from fox know that when Obama came into power, bush left out a 1.2 Trillion dollar / year debt and a financial crisis on top of it. All based on the conservative policies that D Souza camp argue for. 

I don’t understand the difficulty that the conservatives have in understanding Obama’s motives as he is well-meaning to the whole world except conservatives. All i can tell is America has never been or wanted to be imperialistic, pro-colonial(American Indepence from British, Hey?) and always has been a entitlement nation(Obama-care won’t make it any different. Trivia - this was first put forward by a conservative republican candidate). 

P.S At-least Obama was not caught in a private dinner video talking how he will not be the president of those 53% Americans who want all these tax cuts. Romney did call 47% of America as lazy and he won’t care about them. Now it could just be gaffe, but I couldn’t resist bringing it up.

Links:

http://en.wikipedia.org/wiki/2016:_Obama’s_America

http://en.wikipedia.org/wiki/Dreams_from_My_Father

http://www.dailykos.net/archives/003167.html

http://en.wikipedia.org/wiki/Socialism_in_India

http://reason.com/archives/2006/06/06/the-rise-and-fall-of-indian-so

http://en.wikipedia.org/wiki/India_Unbound

http://en.wikipedia.org/wiki/Independence_Day_(United_States)

http://www.youtube.com/watch?v=PyJ_k2z-nTA

http://en.wikipedia.org/wiki/Hosni_Mubarak

http://en.wikipedia.org/wiki/List_of_countries_by_military_expenditures