Things that make my head hurt

Fixing Service Performance with PProf in Go

Fri, 01 Mar 2019 00:00:00 +0000

I tweeted the other day about how I managed to reduce the CPU consumption on one of my services by 90% by removing a single line of code.

One of the things I did not do, mainly since it is difficult to make a full explanation in 256 characters is explain how I identified the problem and the process I took to locate the root cause. This short post sets out to rectify that, and I hope it serves as a useful resource to save you all from repeating my embarrassing mistake.

The Service

The service I was working on was a simple API aggregation service; this exposes a public HTTP API consumed from a ReactJS website. It interacts with two other upstream services, a cache which uses gRPC for the transport and a face detection service which is using Matt Ryer and David Hernandez FaceBox.

I make no apologies for the implementation details of this service; it is not the model on an excellent microservice in fact there are components which really should be delegated out into other services. What I was building was a simple system which would allow me to demonstrate how to use the Consul Connect service mesh and Envoy’s reliability and observability features. If you would like to take a look at the source code, you can find the link on GitHub: https://github.com/emojify-app/api.

The Problem

The service itself was functioning fine, the latency was low, and there were no errors, but the CPU consumption did feel a little high for the traffic received and the work the service is doing. Running this on my Kubernetes cluster it was easy to miss this, but when running this in an environment with lower resources it was a problem as the CPU consumption was starving the other services. This unusual behavior caused me to start to take a look and investigate.

API Service CPU Consumption

The above chart is showing the CPU consumption from the service; this might be nothing normal; looking at something like CPU without context is not the best way to draw a conclusion. What started to look unusual was when I combined the CPU chart, an understanding on the actual work the service was doing at the time which was streaming files over a gRPC connection from another service, and the limited number of requests.

API Service Requests per Second

Something looks fishy here I have a hunch that there is something not quite right as I have built many Go based services and generally they are terrifically efficient on their CPU and memory consumption. Let’s take a look at the upstream that the API is calling and see how that is performing as this service is responsible for sending the files.

Cache Service CPU

There is a dramatic difference there; the Cache is using 10% of the CPU of the API for the same number of requests. This Cache service is reading from a file and sending the bytes of data as a gRPC message. The API service receives that message and writes it as an HTTP response. There is nothing complicated going on there; there should not be such a difference in the two services.

Investigating the Problem

I was now pretty confident that there was a bug somewhere in the API service which was causing it to consume way too much CPU and I needed to investigate. Luckily for me Go has an excellent tool called pprof https://golang.org/pkg/net/http/pprof/ which allows you to inspect the internal working of your application, you can see incredible detail like timings for memory allocation and the execution time for individual blocks of code. Adding this to your code is also incredibly easy, so I decided, I would deploy a new build of my service with the diagnostics included so that I could run a profile.

To enable profiling, I only had to add a couple of lines of code; the first was to import and enable the pprof package.

import    _ "net/http/pprof"

If you already have a web server in your application, then pprof automatically attaches itself to http.DefaultServeMux enables the API at the path /debug/pprof/

I was not using the DefaultServeMux in my application as I am using the Gorilla Mux package for my http handlers. Because I was using Gorilla, I had to add another line of code to enable the HTTP routing to pprof

r.PathPrefix("/debug/pprof/").Handler(http.DefaultServeMux)

After building and re-deploying my application with the instrumentation enabled I could then sample the running processes using the pprof tool. Again this is a straightforward process; the pprof has excellent documentation on how to collect the different profiles, CPU, heap, blocking, etc.

I point the pprof tool at my profile endpoint which is: https://myservice/debug/pprof/profile?seconds=5 this collects a five-second profile of the CPU. The short profile is fine for my requirements as I know the requests are completing quickly, I don’t know why they are consuming so much CPU.

$ go tool pprof "https://myservice/debug/pprof/profile?seconds=5"
Fetching profile over HTTP from https://myservice/debug/pprof/profile?seconds=5
Saved profile in /home/jacksonnic/pprof/pprof.emojify-api.samples.cpu.006.pb.gz
File: emojify-api
Type: cpu
Time: Mar 1, 2019 at 5:09am (UTC)
Duration: 5s, Total samples = 40ms (  0.8%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

Once I had the pprof tool running the first thing I like to do is to view a visual call trace of the collected profile, this allows me to zoom in quickly to the source of the problem. You need to have Graphviz installed to do this, but by merely executing the command pdf pprof outputs a graphical overview of the profile.

The output looked like this:

Immediately I can see that writeString in the protobuf package is consuming a considerable amount of the CPU. Next step is to trace this back up to the source, in my code to understand why.

Following the trace, I finally get to code I have written and saw this:

The source of the problem seems to be a fmt.Println statement, time to dig into the code and see what that is doing and why it is even there.

d, err := c.cache.Get(context.Background(), &wrappers.StringValue{Value: f})
fmt.Println("err", d, err)

So it seems that when I was debugging my application, I was writing the response from the gRPC cache service which is a protobuf message to StdOut. Like a good developer, I forgot this was there and then deployed the application. Serializing this protobuf message in this way was incredibly expensive, annoyingly it was also very unnecessary.

Fixing the Service

The fix could not be more straightforward, delete the line of code, rebuild and deploy the service again, with the new service deployed there was an immediate impact to the CPU consumption.

CPU post fix

Over a 90% reduction in CPU, and what looks like far more normal operating conditions for the service with this traffic. It is good to double check; however, so again I ran a profile, this time the profile showed the hot spot as syscall. The result this time is what I would expect as the service is mainly reading and writing to a TCP socket.

CPU profile after fix

Summary

I always say no experience is bad if you can learn something from it, I certainly learned that I am prone to making stupid mistakes, but it was also fun to dig into pprof again. The whole process of finding the problem and fixing it took me approximately 30 minutes, this, of course, could have been so much longer had the issue not been so pronounced. However, it highlights just how amazing the tooling in the Go ecosystem is.

If you want a takeaway, I have two:

Code review, a fresh pair of eyes would probably spot the unnecessary Println statement
Profile your services before major deployments, it does not take long, and a quick eyeball of the results can save embarrassing mistakes

If you would like to see more detail on the before and after traces, you can download a PDF from the following links:

Before: profile001.pdf
After: profile002.pdf

One final thing, remember to remove the instrumentation from your service, especially if this is running in production. I may or may not have just done this.

Have fun,

Nic

Travelling in China

Sun, 14 Oct 2018 00:00:00 +0000

China is a beautiful place to visit, rich in culture and charm. For the western traveler, it may feel intimidating on your first visit as there are subtle differences to what you may be used to. In this blog, I hope to set out some tips and tricks which will make your trip a success.
Disclaimer: I am not a seasoned traveler in China only having had two fantastic trips to the county. The items detailed below are not a comprehensive overview, but they have made my time in China far more enjoyable.

Visa

The first and most important thing to note is that you are most likely going to need a Visa. If you are planning on attending a conference such as KubeCon China, then you will need a business visa. For tourism, a standard tourist visa will suffice. The difference between the two is that a business visa requires a letter of invitation from the company or conference you are attending. There is no one size fits all for this letter the details needed on it will depend from country to country. Details of the invitation letter will be provided by your local embassy or visa agency. Typically this includes items like full name, passport number, and expiry, home address, date of birth and gender. The letter will also need to carry the official stamp from the authorized company and the signature and contact details of a company representative.

It may be tempting to cut corners and just apply for a tourist visa instead of a business visa; personally, I have done this in the past with no problem. However, if you are questioned at the point of entry you may be refused entry, my advice is, do not risk it and obtain the correct permit. Visas also come in two different types, single entry and multi-entry, for the UK, both types allow you a stay not exceeding 90 days. Often when applying for a single entry visa, you will be granted a multi-entry anyway. Both types are valid for two years. One thing to note is that you can only hold one Chinese visa at any time, if you already have a tourist visa and need to upgrade to a business visa then you will need to write a letter explaining why you wish to surrender and cancel your existing visa. Business visas can be used for tourism but not the other way round.

The visa application form is relatively straightforward however you will need to make a visit to your local embassy for an interview, you will also have to leave your passport with the embassy while they process your application. Personally, I have always used an agency such as CIBIT in London who have managed this process for me, they will also ensure that the details on your form are correct before submitting it to the embassy. Using an agency does carry an additional cost, but I think it is well worth it.

Airports

Airports around the world are relatively homogenized, and it is no different in China, you will find the usual shops and coffee shops and long queues at immigration and security. From an immigration stance upon arriving you will need to complete a landing card (remember to pack a pen). This card comes in two parts one which is submitted on landing and one which you must present upon departure. Keep your departure card safe as failure to provide this when you leave the country may cause problems when you next return to the country and could result in you being denied re-entry.

Border patrol will check your passport and visa, you will be required to provide an electronic copy of your fingerprints, and your passport will be stamped. Do not panic if you are taken to one side, often this is just a case of double checking with another border patrol member. Concerning language, do not expect the border police to speak English, this is generally not a problem, and I have always found the process efficient and without issue. All airports will have instructions and signage in both Chinese and English so navigating is never too much trouble.

Getting around

Airports are generally situated 20+ KM from the center of the city very similar to how things are in the west. There are sometimes public transport links to the center however after a 10+ hr flight the best option is often to take a taxi. In general, the infrastructure for public transport is incredibly good, there has been a massive infrastructure investment in China over the last 10 years in this area, and it is incredibly good value.

Taxis

Taxis are cheap and plentiful, you will find a huge number of taxis at train stations and airports. You will also find a large number of unsolicited drivers offering to take you to your destination. In general no matter which country you visit, I find using an official taxi the safest option. Just follow the sign to the official taxi rank, there will be one in English and you will find it. Most likely you will also find a large queue of people waiting but the lines clear pretty fast.

Taxis are not particularly expensive, a trip between Shanghai Hangdong airport and a hotel in the Bund, which is a trip of about 45 minutes (depending on traffic) will cost approximately 100 yuan (about 10 euro). Do not expect the taxi drivers to speak English so ensure that you have the location of your hotel on your phone in Chinese which you can show the driver. All licensed taxis run a meter, and this usually starts ticking up from about 14 yuan.

Tipping does not seem to be the norm and can cause some confusion when you refuse your change. Once the message does get through the gesture is always welcomed.

Uber is not present in China, there is a local alternative Dede; however, I have not used this, my advice is if you do not have a written destination, then ask the hotel or restaurant to request a taxi for you and ensure they tell the driver where you wish to go to. Most taxi drivers are honest and friendly people, especially when ordered from an official taxi rank or hotel.

Should you hail a taxi from the street, do not be alarmed if you find a stranger in your cab. Ride sharing is a thing in China, and some drivers may opportunistically attempt to double their fare by operating in this way. This will not happen when you request a car from a hotel or take it from a train station.

In general, taxis are in plentiful supply, and will generally only accept cash or mobile payment, I have not seen a taxi where it is possible to pay by credit card, so ensure you have enough currency on you to cover your trip.

Busses

As previously mentioned the public transport infrastructure in China is incredibly good, and bus routes are everywhere. Taking a bus is very cheap with a single trip costing about 2 yuan. In the major cities, you will not be able to pay for the bus in cash, mobile payment is everywhere, and this seems to be the only way to pay for a bus. Unless you are adventurous, my advice is to skip the bus and grab a cab.

Trains

The trains in China are both efficient and cheap, they are also a great way to get between cities. You can take the bullet train between Shanghai and Beijing in around 4 hours. The train is comfortable, fast and a wonderful way to see the country which you would completely miss if flying. Trains are also a fantastic way to travel shorter distances, a trip by train between Shanghai and Hangzhou, a distance of around 100 mile takes only 1hr and costs approximately 10 euro. The simplest way to purchase tickets for trains is to use Trip.com or even better the Trip.com app. This app or website will show you the full train schedules and will allow you to purchase your tickets using a western credit card.

Once you have purchased your tickets on trip.com, you will need to pick them up from the ticket office in the train station, the location of the ticket office usually is outside the central station before security. Sometimes the ticket office will have an exclusive line for “Foreigners”, however do not expect the person at the booth to speak English. Collecting your tickets is, however, a pretty simple process, Trip has a particular screen which you show to at the ticket office, you will also need your passport. Hand both of these items over at the counter, and you will get back your ticket. The ticket will have the number of the train, and time of departure printed at the top, it will also have your carriage and seat number printed in the top left. To board the train, you will need to enter the central part of the station, and this is generally going to require that you show both your ticket and passport to security. You will also have to put your luggage through the Xray machine similar to the experience at an Airport. While this process is far quicker and nowhere near as thorough as Airport security the lines can be quite long at peak times. Once through security you need to find your platform, the train departure boards in China are shown in both Chinese and in English. Find your train number on the board and head to the gate (airport style) which is associated with that number.

Train boarding generally starts about 10 minutes before departure and ends 5 minutes before departure so leave yourself plenty time to check in. People will also begin to queuing early at the gate, and the gates will support multiple trains sometimes with only 10 minutes difference in departure. You will be able to tell when it is time to board as your train number will show up green at the top of the board. Again you will need to pass the barriers to get to the platform, this is a simple process, put your ticket in the gate, walk down to the platform and when the train arrives, get on board. Trains in China are incredibly punctual, the seats are comfortable, they are fast and an excellent way to see the countryside as you hurtle along at 300KM/h.

Metro

Chinese cities will generally have a metro system, the metro in the larger cities is very similar to what you may find in European cities with many different lines allowing you to get pretty much anywhere. Like European cities, the metro is crowded during peak times. Tickets are cheap costing roughly 3 yuan (0.4 euro) for a single journey and tickets can be purchased from either the automated machines or from the ticket counters in the station. In addition to single tickets, you can also buy day and week-long travel cards. Disclaimer here, on my most recent trip I did not take the Metro, I mainly took a Taxi or walked to my destination. I, however, plan to take the metro when I return to Shanghai in November and will update my blog then.

Walk

Walking is always a great option when exploring, take advantage of the slower pace, look around (not at your phone), and enjoy. Most cities can, however, be crowded to expect a reasonable amount of human traffic. If you are used to a city like London, then there is nothing really to note, crossing the road is best done at the pedestrian crossings however in China, there is a considerable number of electric mopeds, expect these not to stop when you are crossing the road. They do have to stop and will not deliberately run you over but just be careful before you step into the street.

For navigation I found Google maps really reliable, that said, Google is mostly banned in China, but there are many ways around this, check out the “Connecting to the Internet” section of this post to ensure you can get access to maps.

Currency

The currency in China is the Renminbi (RMB, peoples currency), you will find things generally referred to as “Yuan” which is also a basic unit of RMB. This is similar to the UK where the currency is Sterling, but the units are Pounds and Pence. Notes and coins come in sizes from 1 yuan (1/10th of a euro) to 100 yuan (10 euros).

There are cash machines everywhere where you can obtain money using your cash card and pin number, most of these will have an option for transacting in English, and they work exactly the same as you would expect anywhere in the world. One thing to note is that every now and again you may find a cash machine which will not work with foreign cards, the best advice is to use Hotel cash machines or major Chinese banks.

It is a good idea to always carry a small amount of cash with you, most places will readily accept credit cards however there is a movement towards mobile payment, and some stores (even large chains) will not accept credit/debit cards. Taxis are generally happy with cash however again beware that they may be more used to receiving mobile payments and may not have change. Another thing to note is since taxi drivers rarely speak English they may not be able to tell you they do not have change. My advice here is to just round up your fair as a tip. While tipping is not prevalent, it is always appreciated.

Mobile payments are sweeping China by storm, most shops, big and small, taxis, railway stations, ok, everywhere accept them. The two primary sources are WePay and AliPay. WePay is built into the WeChat app (China’s whats app), it is a wallet style where you top up your WePay wallet with currency from a credit or debit card. The rub for Foreigners is that while you may be able to add your card to WePay, do not expect it to allow you to top up. There are two ways you can get around this, the first is to ask a local colleague or friend to send you a Red Letter, which is WeChat’s way of transferring currency between users. The second is you can use a service such as vpayfast.com, using this service you will be able to top up using your credit or debit card, and the service will send you a red letter for the amount through WeChat. They will charge you 8% of the transaction as a fee, but this has been the only reliable way I have found to top up WeChat. I think it is worthwhile to add 200 yuan to your WeChat wallet while in China, you never know when it will be useful, remember some small stores might not even accept cash.

In my personal experience, I find my Monzo account works really well across the globe with favorable transfer rates and the convenience of being able to see what you are spending when you spend it. Another option which I am planning on trying on my next trip is Transferwise, with a Transferwise account you can hold local currency and spend this with the convenience of a master card debit card.

Any major credit card Visa or Mastercard should be accepted except American Express which you may find you can only use in major hotels. Like traveling anywhere having a backup plan is always a good option.

Shopping

Like the rest of the world, walk down a shopping street in China and expect to find all of the usual brands. Starbucks is everywhere, Zara, and all the other usual suspects, you will also find local brands. To be honest, I have not really done a great deal of shopping so I can’t really comment on prices.

The only thing I will note is to bring your attention back to something I mentioned in the currency section. Do not be surprised if the store will only accept cash or mobile payments. I experienced this when I went to buy a gift for my Nephew, the store was a busy modern chain called Line Friends. The item in question a Teddy Bear, not especially cheap at 350 yuan. I was surprised to find that when trying to pay that they would not accept my credit card and I was directed to mobile pay (which I could not use due to top up problems). The cashier did not speak English (and why should we expect them to, it is not an English speaking country), and eventually figured out that they would accept cash.

I can not confirm this 100%, but from speaking to my Chinese colleagues, it appears this is not unusual, as everything seems to be headed to the mobile pay route. The key takeaway, do not expect to be able to use your credit card everywhere, cash is always a good bet, but you may find this is not accepted in some smaller stores. Mobile payment is king.

Food

Food in China is delicious, I may be biased as it is one of my favorite international cuisines but my recommendation is to embrace this and take every opportunity you can do dine like a local. The menus are probably not what you will see in Chinese restaurants in the West as these have been primarily homogenized, but the dishes are not so different.

Hairy leg crab

Hairy leg crab is a delicacy, found in the late summer, the crabs spend their life in the paddy fields where their presence helps the rice grow. Then unfortunately for the crab and enjoyment of diners, it is time for the dinner table. Disclaimer: while I embraced local food and rarely eat anything else when in China, I have a general rule where I will not eat anything with a face.

In addition to local foods, you will also be able to find the usual suspects, Starbucks is everywhere, Macdonalds on most corners but do your best to avoid these and embrace the local culture. A meal for two people in a modest restaurant will cost you less than 200 yuan which is excellent value. Compare this to a double espresso in Starbucks (roughly 20 yuan) or a double shot skinny latte (35 yuan), and it seems excellent value. My main benchmark for the price of food in a country is often the Big Mac meal, in China, this is roughly 35 yuan or 3.5 euros.

You will find that menus are often only written in Chinese but will have pictures, there will usually be English translations or photographs for food in the larger cities. Again do not expect the restaurant staff to either speak English and you may have trouble trying to pay with your western credit card. Cash is generally the best option and always a good backup.

The menu will differ depending upon the region you are in, and it is going to be rare to find vegetarian food as almost everyone is an omnivore.

Vegetarians beware, while you will find many vegetable dishes on the menu, often these are prepared with either meat or fish stock. Large hotels might be your best bet if you do not eat meat or fish, they will be best to cater to your needs.

Language

For the traveler who is used to Europe do not expect Chinese people to speak English as a second language. Hotels will almost always have staff able to speak English but shops, taxi drivers, restaurants, and even Starbucks will most likely not.

Also, remember that the Chinese alphabet (Pinyin) is not going to be easily readable as each symbol is a sound rather than individual letters forming works. Many larger establishments, train stations, airports, and road signs will be written in both Chinese and English which makes navigation easy enough, but it is not prevalent.

A tip for taxis is always to have your destination written in Chinese which you can show to the driver on your mobile phone. Google translate is also a potential option. The best bet when getting a taxi from a hotel is to ask the concierge to request a cab for you and to ensure that the driver knows the destination.

Learning Chinese for a short trip is probably not going to be an option for you, however, no matter where I travel I find it is respectful to at least be able to say Hello, Goodbye, Please and Thank You in the local language:

Hello - Ni Jiao
Goodbye - Zai Jian
Thank You - XieXie (c c)
Please - Qing (ching)

There are many excellent options for learning the language Duolingo has a great free course which I highly recommend.

Safety

I am unable to comment on what the level of crime is like in China, what I can say is that on my two trips I have never felt unsafe. There is a high presence of Police in most cities and train stations and pickpocketing, or petty crime does not seem to be a problem. That said, apply the rules you would in any city, keep your purse or wallet secure on your person and do not leave your belongings lying around.

There are generally plenty of people about even late in the evening, and my personal recommendation is to take the same level of care you would if you are traveling to London. I felt completely at ease walking around China, and as a 2m tall white guy I certainly stand out in the crowd. Occasionally you will be approached by beggars, but this is really rare even in a city as big as Shanghai. They are never threatening and will leave you alone eventually, but if you do have a little change, it will be very well appreciated.

Like any unfamiliar country it takes time to adapt to your surroundings, but after a couple of days, you will be navigating like a native. Do not worry, be sensible and you will have a fantastic time.

Connecting to the Internet

Ok techies, this is probably the thing most important to you, connecting to the internet either on your phone or computer is going to be essential to a good trip. The great firewall will stop you from accessing things like Google search, Google maps, and Gmail; however, this can typically be circumvented by using a VPN. In the rest of the world I am a big fan of Tunnel Bear; however, I found this somewhat patchy in China. I recommend you try ExpressVPN, this will work on your laptop, iDevice and Android platforms. My recommendation is you to set this up before you leave as some of the download sites can be blocked once in the country.

China has first-class mobile broadband, if you think people in the West are obsessed with their phones, China takes this to a new level. I am on Vodafone in the UK and using my plan in China costs an additional 6 euros per day which is well worthwhile. Interestingly I found that Google services would mostly work from my iPhone while I was roaming on the 4G network. I am not sure if this is because the system knows I am a foreigner. In general however even when on a mobile network I find that running ExpressVPN is your best chance to ensuring a reliable connection to all my Google services.

Summary

China is a beautiful country and the Chinese are wonderfully warm and welcoming. The food is fantastic, the infrastructure makes it so easy to travel around, and the scenery breathtaking.

Don’t be put off by comments like China is so polluted, China is so busy you cannot move, in my experience,, this is over exaggerated, I live in London where we have some of the worst air quality in the world, and as for congested, we take things to the next level. China is making amazing inroads to improve the quality of life in the cities. You will not find more electric vehicles anywhere on the planet, the train network is world class, yes there are traffic jams, but there are traffic jams in any large city.

As for the pace of life, is China, frantic, fast moving, exhausting? The pace of life moves at a speed you want it to, you can have a wonderfully relaxing time, just run at your own pace. Embrace the culture, drink tea, not coffee, eat noodles not a burger, enjoy the juxtaposition between rich history and modern metropolis. I hope this article helps and the pleasant memories from your trip last a lifetime.

Controlling your Holiday Lights with Raspberry Pi, Alexa and Go

Sun, 24 Dec 2017 00:00:00 +0000

In this post, we are going to take a look at how we can control the lights on your tree using the Amazon Alexa and a Raspberry Pi.

Why?

Two reasons, firstly because we can, and secondly there is an unusual architectural pattern which needs to be addressed when attempting to bridge the gap between your home and the internet. How does the IOT device running in a NAT’etd environment receive messages from the outside world.

Disclaimer: The code is not necessarily production worthy, it might work through Xmas, there are more clicking on interfaces than I would generally like.

Requirements

To create this system, you will need the following components:

1 Raspberry Pi (Any model with network will do)
1 433MHz TX/RX module for the Pi - Aukru 433 MHz RF Wireless Transmitter + Receiver Module: Amazon.co.uk: Electronics
1 Amazon Echo
Cheap 433MHz radio controlled power sockets- Remote Control Socket Wireless Heavy Duty 13A Adaptor: Amazon.co.uk: Electronics
AWS account
Internet
Some programming knowledge

Architecture

When you issue a command to Alexa “Hey Alexa, ask Christmas tree lights to turn on,”
The language processing in AWS will transform this command into a call to a Lambda function.
The Lambda function adds a message to a Simple Queue Service message queue
The app running on the Pi is reads this message queue
It then transmits a code corresponding to on/off over 433MHz radio which is then picked up by your radio controlled power plug

Using SQS allows us to bypass the fact the Pi needs to be exposed to the internet. The Pi makes an outbound connection to the SQS and sits waiting for messages.

Configuring the Raspberry Pi

The first thing to get started is to configure the Pi and the 433Mhz transmitter. Before we can switch our lights on and off, we need to determine what codes are used by your power sockets. To do this, we are going to attach the receiver portion of the 433MHz radio to the Pi we can then sniff for codes by pressing the buttons on your remote.

The way that the power sockets work is that a binary code is transmitted over the radio. The protocol uses a sequence of radio waves to signify the 1s and 0s in the code. The protocol defines what duration between the highs and the lows and that corresponds to a 1 or a 0 in the code.

Grab your Raspberry Pi and attach the Transmitter to the GPIO pins, you will need to connect the VCC pin on the transmitter to the 5v power pin on the Pi, I am using pin 2 and the GND to the ground pin on the Pi, I am using pin 3. Lastly you connect the DATA pin to one of the GPIO pins, I am using pin 4.

You then need to run the application on your Pi and set it to scan mode; you can find a compiled version of the application in the GitHub repository:

Releases · nicholasjackson/rcswitch · GitHub.

You will need to choose the appropriate version for your PI Arm6 is used for Pi Zeros and first-generation Pi’s and the Arm7 version is suitable for the Pi model 3.

Deep in the code

The code is written in Golang and uses the Periph.io library to interface to the GPIO pins. The first thing we need to do is to initialize the Periph.io library using the Init function.

examples/main.go

    _, err := host.Init()
    if err != nil {
        log.Fatal(err)
    }

We can then get a reference to our pin and start scanning for codes:

    // Lookup a pin by its number:
    p := gpioreg.ByName(*pin)
    setPin(*scan, p)
    log.Printf("%s: %s\n", p, p.Function())

    sw := rcswitch.New(p)

    // scan for codes
    if *scan {
        log.Println("Starting code detection, press a button on your remote control")

        sw.Scan()
        return
    }

We set the pin mode based on whether we want to send or receive codes for receiving we set the pin to input mode with edge detection for transmitting we set the pin to output mode.

// sets the pin to either input or output depending on if we are scanning

func setPin(scan bool, p gpio.PinIO) {
    if scan {
        // Set it as input, with an internal pull down resistor:
        if err := p.In(gpio.PullDown, gpio.BothEdges); err != nil {
            log.Fatal(err)
        }

        return
    }

    if err := p.Out(gpio.High); err != nil {
        log.Fatal(err)
    }
}

Internally the Scan function uses the WaitForEdge feature of the pin, whenever the pin changes state, i.e., low to high. We record the durations between these changes and this allows us to parse against the known protocols and transform it to a string representing the switch message.

My Go library which works with Periph.io can be found here:

https://github.com/nicholasjackson/rcswitch

This library has been ported from a C++ library written by Suat Özgür:

https://github.com/sui77/rc-switch

Detecting codes

Run the example application on your Pi setting the correct pin and scan mode, and we can press some buttons on our remote and capture the codes.

pi@pi04:~ $ ./rcswitch_arm6 -pin 4 -scan true
2017/12/24 13:14:20 GPIO4: In/Low
2017/12/24 13:14:20 Starting code detection, press a button on your remote control

Now press the off button on your remote which corresponds to the power plug you would like to control, hopefully, if your sockets are supported by the library you should see the application output something like the following.

2017/12/24 13:14:23 decimal: 16404 // off
2017/12/24 13:14:23 binary: 000000000100000000010100
2017/12/24 13:14:23 bitlength: 24
2017/12/24 13:14:23 protocol: 0
2017/12/24 13:14:23 
2017/12/24 13:14:25 decimal: 16405 // on
2017/12/24 13:14:25 binary: 000000000100000000010101
2017/12/24 13:14:25 bitlength: 24
2017/12/24 13:14:25 protocol: 0
2017/12/24 13:14:25 

We will need to record the code and the protocol that is output so copy this down for use later on. We can now do the same for the on button, again record this code; the protocol should be the same.

Setting up SQS

Now we have the codes we can go ahead and create an SQS queue for the application in AWS. Go to the AWS console and click on the SQS section.

Click create new queue and fill in the Queue Name Choose Standard Queue Click configure queue and change the message retention period to 1 minute. Click create queue Click Permissions and add two actions

SQS:ReceiveMessage
SQS:SendMessage

Copy the address for the SQS queue from the details tab; we are going to need this for the application which will listen to messages on the queue.

Running the receiver application on the Pi

We can now run our application on the Pi which will receive these codes and use the rcswitch library to transmit the codes.

You can find the source code for the application, including compiled binaries if you do not wish to build it yourself:

https://github.com/nicholasjackson/xmas-lights-v2

Disconnect the receiver from your Pi and connect the transmitter, again the VCC connects to the power on pin 2, GND to pin 3, and the DATA to pin 4.

main.go

    // Setup the GPIO settings for the 433MHz transmitter
    p := gpioreg.ByName(*pin)
    if p == nil {
        log.Fatal("Unable to find pin: ", *pin)
    }

    p.Out(gpio.High)
    log.Printf("%s: %s\n", p, p.Function())

    sw := rcswitch.New(p)

    c := lights.Config{
        OnCode:   *onCode,
        OffCode:  *offCode,
        Protocol: *protocol,
        SqsURI:   *sqsURI,
    }

    // Create a new SQS Vice transport
    t := sqs.New()

    // Create a new instances of the lights application and setup the transport
    l := lights.New(c, t, sw)
    l.Setup()

    fmt.Println("Listening for messages")
    l.Listen()

We again grab a reference to our GPIO pin using the Periph.io library and configure the package with our codes which we captured earlier, the protocol, and the address for the SQS queue.

Internally we are using the excellent Vice library created by Mat Ryer:

https://github.com/matryer/vice

Vice provides a common abstraction for many different queues such as SQS, and NATS.io streaming and provides us with a Go channel which we can listen to for messages.

lights/lights.go

func (l *Lights) Listen() {
    for m := range l.messageChan {
        message := decodeMessage(m)
        switch message.Command {
        case CommandTurnOn:
            l.swtch.Send(l.config.OnCode, l.config.Protocol)
        case CommandTurnOff:
            l.swtch.Send(l.config.OffCode, l.config.Protocol)
        }
    }
}

For Vice to be able to connect to SQS you will need to set some environment variables corresponding to your AWS account.

export AWS_SECRET_ID=xxxxxxx
export AWS_SECRET=xxxxxxx
export AWS_REGION=[YOUR AWS REGION] // i.e. eu_west_1

We can now run the application using all the information provided earlier and go ahead and create our Alexa skill.

./lights -pin 4 -on_code 0000001101010 -off_code 00000110100 -protocol 0 -sqs_uri http://something.com

Setting up the Alexa skill

Open the Alexa development console and click the Get Started button below Alexa Skills Kit https://developer.amazon.com/edw/home.html#/

Click Add New Skill. Give your skill a name and an invocation name.

Click Save, then press Next.

We now need to configure the details for our Skill; this includes the things that the skill will respond to (Intents) and variables which will be extracted from the command and sent to the Lambda function (Slots).

First, click the menu on the left and add a new Slot Type. Give it a name of onoff Next step is to configure the values for the slot, and unsurprisingly these will be on and off, add these values using the UI.

Now add an intent by clicking the ADD button on the left-hand menu, give the Intent a name of SwitchLights. If you change the name of the Intent or the Slots, you will need to change the Lambda and the application code.

We are going to add two sample utterances:

“turn {onoff}”
“switch {onoff}”

The curly brackets tell the skill that we would like to detect values defined in our onoff slot created in the previous step. When you issue a command to Alexa these values will be extracted and sent with the request to the Lambda function.

Before saving make sure that you select the Slot Type by clicking on the slot on the right-hand side and then selecting the Slot Type from the new dialog box.

Your intent should now look like the following example:

Now we can Save and Build our model.

Creating our Lambda

The next step is to create a Lambda function which will be called by the Alexa skills kit. This function is going to be written in Node.js.

Go to the AWS Lambda section of the AWS consul and select create new lambda, we are going to create a lambda using one of the blueprints as we will need the Alexa SDK node module. Click create lambda from blueprint and choose alexa-skill-kit-sdk-factskill.

Click Configure and on the next screen give your function a name. Next we need to create a Role for the function which has permissions to write to logs and also write messages to the SQS queue.

Choose Create a custom role, and the dialog will change to allow you to configure the role.

The default options will be fine however we do need to add the queue permissions, click the edit button and when asked if you have read the documentation nonchalantly click “OK” because nobody reads the documentation.

We need to modify the policy; you should be able to replace the contents of the box with the details listed below. You will need to replace [YOUR ARN HERE] with the ARN of the queue that you created earlier.

IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Action": [
                "sqs:SendMessage"
            ],
            "Effect": "Allow",
            "Resource": "[YOUR ARN HERE]"
        }
    ]
}

When you return to the Lambda console, make sure the role you have just created is selected in the Existing role drop down box and click create function.

Now go down to the code editor and delete all of the code which is in there.

We are going to replace all of the code with our function which is listed below:

handler.js

const Alexa = require('alexa-sdk');
var QUEUE_URL = 'https://sqs.eu-west-1.amazonaws.com/xxxxxxxx/xmaslights';

var AWS = require('aws-sdk');
var sqs = new AWS.SQS({region : 'eu-west-1'});

const handlers = {
  'SwitchLights' : function() {
    //this.emit(':tell','Ok, Merry Christmas')
    
    console.log(this.event.request.intent);
    var value = this.event.request.intent.slots.onoff.value;
    if (value != "on" || value != "off") {
      this.emit(':tell:', 'Sorry I can only turn the lights on and off');
    }
    
    var params = {
      MessageBody: JSON.stringify({command: value}),
      QueueUrl: QUEUE_URL
    };
    
    const self = this
    sqs.sendMessage(params, function(err,data){
      if(err) {
        self.emit(':tell', 'Sorry, something went wrong, maybe you are not a dope coder after all');
      }else{
        self.emit(':tell', 'Ok, Merry Christmas');
      }
    });
    
  },
  'Unhandled': function() {
    this.emit(':ask', 'Sorry, I dont know how to do that');
  }
};
  
exports.handler = function(event, context) {
  const alexa = Alexa.handler(event, context);
  alexa.appId = "amzn1.ask.skill.xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx"
  alexa.registerHandlers(handlers);
  alexa.execute();
}

You will need to replace the placeholder QUEUE_URL with the address of your SQS queue that you created earlier. You also need to change alexa.appId value and set that to your app id. You can find the app id in the grey bar at the top of the developer console.

Save your function.

We can now go ahead and test our function, all of the parts should be in place to send a test request which will send a message which will be picked up by our Raspberry Pi.

Click test and configure a new test message which corresponds to the listing below:

{
  "session": {
    "new": true,
    "sessionId": "SessionId.04916bd6-7f39-49fb-8860-d8b0ae4b7465",
    "application": {
      "applicationId": "amzn1.ask.skill.fb690610-84f3-4e8f-b38c-d13f727195a9"
    },
    "attributes": {},
    "user": {
      "userId": "amzn1.ask.account.AFQHEN5Q3F5JEM6ZB2SGUQ6IQTRVKV75SNENKTWYRBA2OD4QL427QLHEEST7GI2GQ42WDLJUEAYTNXXXXX4K27YZCAMYTJCIQGZV2ZRIHFWZ7BMRTI7XXXXXXFN2F7LKCM2ED6NZDGL6MYVWOQJVTUEMYIRHKHVGFXKOV3XVMILXZVN2WJARPLKCQPFXQ6QGQ57RIHY"
    }
  },
  "request": {
    "type": "IntentRequest",
    "requestId": "EdwRequestId.cfdaab4a-af02-4d76-be6a-e235c4323072",
    "intent": {
      "name": "SwitchLights",
      "slots": {
        "onoff": {
          "name": "onoff",
          "value": "on"
        }
      }
    },
    "locale": "en-GB",
    "timestamp": "2017-12-23T16:18:21Z"
  },
  "context": {
    "AudioPlayer": {
      "playerActivity": "IDLE"
    },
    "System": {
      "application": {
        "applicationId": "amzn1.ask.skill.fb690610-84f3-4e8f-b38c-d13f727195a9"
      },
      "user": {
        "userId": "amzn1.ask.account.AFQHEN5Q3F5JEM6ZB2SXXXXXXRVKV75SNENKTWYRBA2OD4QL427QLHEESTXXXXXX2WDLJUEAYTN7W5VGTXPSZ4K27YZCAMYTJCIQGZV2ZRIHFWZ7BMRTI7DGUA5P5N7FN2F7LKCXXXXXGL6MYVWOQJVTUEMYIRHKHVGFXKOV3XVMILXXXXRPLKCQPFXQ6QGQ57RIHY"
      },
      "device": {
        "supportedInterfaces": {}
      }
    }
  },
  "version": "1.0"
}

All being well when you press test, your switch should have turned on, if you look at the output in the log from the application running on the Raspberry Pi you should see that it has received a message and has transmitted our code.

2017/12/24 15:09:26 Starting lights: v0.1
2017/12/24 15:09:26 GPIO4: Out/High
2017/12/24 15:09:26 Listening for messages
2017/12/24 15:10:12 Received Message: [123 34 99 111 109 109 97 110 100 34 58 34 111 110 34 125]
2017/12/24 15:10:12 Sending turn on with code: 000000000100000000010101
2017/12/24 15:10:12 Transmitting: 000000000100000000010101

Select Publish a new Version from the Actions menu; then we need to add a trigger.

Back at the top in the designer section, we need to add a trigger for Alexa, select the Alexa Skills trigger from the menu on the left and then configure the options which allows Alexa to call your function.

Finally, copy the ARN for the lambda which will look something like this: arn:aws:lambda:eu-west-1:38390000802431:function:xmaslights2:11 and go back to the Alexa skills developer portal.

Configuring Alexa with Lambda

If you click on the configuration menu option, you can paste your Lambda’s ARN into the box.

Click the next button, and we should be able to test our setup. In the test box type an example command like the following:

"Ask Christmas lights to switch on."

You should see a successful response in the output window, and the remote switch should enable. To switch the lights off, we can use a corresponding command”:

"Ask Christmas lights to switch off."

Assuming all of this is now working the last step is to check the box which enables testing of your function, and we can add the skill to our account and try it with our Echo.
If this box is not available, make sure you have linked your Echo device with your Developer account. https://developer.amazon.com/docs/custom-skills/test-a-custom-skill.html#h2_register

All together now

The last thing we need to do is to give it a test; you should now be able to say to Alexa.

"Alexa, ask Christmas lights to switch on."

And the result, instance festive cheer.

Summary

This walkthrough has shown you how you can create a simple Alexa skill and how you can link this to an IOT device such as the Raspberry Pi. Now when dragging your self off to bed after far too much Christmas turkey, you no longer need to find the remote buried under piles of wrapping paper just command your faithful robot:

"Alexa, ask Christmas lights to turn off."

Happy holidays

Resources

Source Code
https://github.com/nicholasjackson/xmas-lights-v2

RCSwitch Go Libary for Periph.io
https://github.com/nicholasjackson/rcswitch

433MHz Switch Protocol Info
https://github.com/sui77/rc-switch/wiki/KnowHow_LineCoding

Building a Slackbot with AWS Lambda and Lex

Tue, 25 Apr 2017 00:00:00 +0000

This post is intended to walk you through creating a Slackbot with AWS Lambda and Lex. I am not using any of the Lambda frameworks like Serverless or Apex in this post instead demonstrates how it is possible to create a Lambda function with pure Node and deploy using HashiCorps Terraform. The simple application is a bot which listens for messages that correspond to the phrase “Which timezone is [user] in?” it then calls the Slack API and retrieve the timezone for that particular user. Lex has the capability of defining an Intent which is a request; the natural language processing means that we do not need to cater for every permutation of the question such as:

Which timezone is nic in?
What timezone is nic in?
Hey @otheruser what timezone is nic in? The Lex runtime selects the best intent based on its natural language processing. In addition to this, we can define variables or Slots which Lex extracts out of the intent and pass to the function.

You can find the source code for this post at https://github.com/nicholasjackson/slack-bot-lex-lambda:

Step 1: Creating the Lambda function

To add interactivity to Lex we can pass an intent to a Lambda function, in our example, this function is going to call the slack API and fetch the Timezone that matches the given user.

file://src/index.js

If we take a look at src/index.js in the code repository, we see that it is a relatively simple Javascript lambda handler.

/*jslint node: true */
/*jslint esversion: 6 */
'use strict';
04
process.env.PATH = process.env.PATH + ':' + process.env.LAMBDA_TASK_ROOT;
06
var slackClient = require('slack');
var dispatcher = require('dispatcher');
09
// --------------- Main handler -----------------------
11
function loggingCallback(response, originalCallback) {
  originalCallback(null, response);
}
15
// Route the incoming request based on intent.
// The JSON body of the request is provided in the event slot.
exports.handler = (event, context, callback) => {
  try {
      // By default, treat the user request as coming from the America/New_York time zone.
      process.env.TZ = 'America/New_York';
      console.log(`event.bot.name=${event.bot.name}`);
23
      dispatcher.dispatch(event, slackClient, (response) => loggingCallback(response, callback));
  } catch (err) {
      callback(err);
  }
};

The first thing to note is line 5 because we are going to be bundling a couple of node modules with our Lambda function we need to configure the environment path to be able to pick these up. The path only needs to be set once and not on every file. We are also importing a couple of local modules slack which is our slack client and dispatcher which encapsulates the logic for processing the Lex request. These have been separated out into modules to allow individual unit testing.

line 24 is where we are calling the dispatcher to process the request, we pass to this the reference to the slack client, again separating dependency and increasing testability.

file://src/dispatcher.js

Looking at the file src/dispatcher.js we export a single function which is the one we are calling from our main handler.

/**
* Called when the user specifies an intent for this skill.
*/
exports.dispatch = function(intentRequest, slackClient, callback) {
console.log(`dispatch userId=${intentRequest.userId}, intent=${intentRequest.currentIntent.name}`);
 89
const name = intentRequest.currentIntent.name;
 91
if(name == 'WhatTimezone') {
   const user = intentRequest.currentIntent.slots.User;
 94
   slackClient.getUserDetails(user, (ret) => {
    var message = { contentType: 'PlainText', content: '' };
    var outcome = 'Failed';
    
    if (ret.err === undefined) {
      outcome = 'Fulfilled';
      if (ret.timezone !== undefined) {
        message.content = user + ' is in ' + ret.timezone;
      }else {
        message.content = 'Sorry I can\'t find user: ' + user;
      }
    }else {
      console.log("error: " + ret.err);
    }
    
    callback(close(
      null,
      outcome,
      message,
      null));
  });     
116
  return;
}

  throw new Error(`Intent with name ${name} not supported`);
};

The request that is sent to the function from Lex looks something like this:

{
  "currentIntent": {
    "slots": {
      "User": "nicjackson"
    },
    "name": "WhereIs",
    "confirmationStatus": "None"
  },
  "bot": {
    "alias": "$LATEST",
    "version": "$LATEST",
    "name": "SlackBot"
  },
  "userId": "John",
  "invocationSource": "DialogCodeHook",
  "outputDialogMode": "Text",
  "messageVersion": "1.0",
  "sessionAttributes": {}
}

As we mentioned in the first section, Lex has the capability to extract information from the request and to include it in slots passed as an object to your function. The first thing we are doing in our function is to determine if this Lambda function can handle the intent. On line 92 we are checking this; it is not necessary to perform this check however it could save some headaches if you accidentally wire the wrong function to an Intent. We then call the slack client on line 95 and pass a callback which crafts the response to send back to Lex. The main part of this code to note is line 110, in this instance, we are sending back a Close response to Lex, this sends the message that the response is final and no further interaction takes place for this session. Other valid values are ElicitSlot and ConfirmIntent, the example code has helper functions which can craft the correct responses however for the purpose of simplicity we are only using the close function in this walkthrough.

function close(sessionAttributes, fulfillmentState, message, responseCard) {
  return {
      sessionAttributes,
      dialogAction: {
          type: 'Close',
          fulfillmentState,
          message,
          responseCard,
      },
  };
}

sessionAttributes are what Lex uses to pass between client and code as it does not log and store these values. A responseCard if used is passed back to slack and can be used in addition to the message to request further information in the form of lists to prompt the user for action. For example, in our application, if we do not find the user mentioned we could send back a list of possible matches, and the user could then click on this item rather than having to type the response. fulfillmentState is a string variable which contains the options of “Fulfilled” or “Failed”. Moreover, the message is an object with the syntax:

{
  contentType: 'PlainText',
  content:     'Some content'
}

file://src/dispatcher.test.js

As an investigation into the workflow for building lambda functions with JavaScript, I have configured a couple of tests for the dispatcher which is the most complicated part of the function.

describe('dispatch', () => {
17
test('calls the slack API with unknown user', done => {
  var mockSlack = {
    getUserDetails: function(user, callback) {
      callback({ });
    }
  };
24
  dispatcher.dispatch(intentStub, mockSlack,(outcome) => {
expect(outcome.dialogAction.fulfillmentState).toEqual('Fulfilled');
    expect(outcome.dialogAction.message).toBeDefined();
expect(outcome.dialogAction.message.content).toEqual('Sorry I can\'t find user: nicjackson');
    done();
  });
  
});

Using dependency injection on our dispatcher, we can isolate the slack client giving us the capability to unit test the dispatcher. Jest provides some nice matchers for the assertions and if needed also has a comprehensive mocking framework. These tests are by no mean comprehensive or fully covering but they do indicate that it is possible to write Lambda TDD style, and with good test coverage can make up for the inability to easily run locally. The serverless framework does provide the capability to run in dev mode, but my opinion is that by removing this crutch you are pushed to producing a high-quality test suite which carries far greater benefit with long term maintenance.

file://src/slack.js

Our slack client is fairly straight forward and makes a call to the users.list endpoint on the slack API, iterates over the list and selects the timezone for the matching user. At the time of writing, there was no way to directly retrieve a slack user using the API with only having the name of the user, not the ID.

/*jslint node: true */
/*jslint esversion: 6 */
'use strict';
04
var slackClient = require('@slack/client').WebClient;
var token = process.env.SLACK_API_KEY || '';
07
var sc = {
};
10
exports.getUserDetails = function(username, callback) {
var web = new slackClient(token);
web.users.list({}, function(err, info) {
   if (!err) {
     var timezone = { };
16
    info.members.forEach((user) => {
       if (username === user.name) {
         timezone = { timezone: user.tz_label };
       }
     });
     
     callback(timezone);
   } else {
     callback({ error: err });
   }
});
};

The main thing to note in this function is that we need the slack API token which is set in the Lambda setup when we deploy the function.

Step 2: Deploying the Lambda function

To deploy the lambda function uses Terraform by HashiCorp, the config for the deployment is found in the terraform folder. For this, to work, we need valid AWS credentials and the Slack API token set to an environment variables on your machine. I recommend using direnv and a .encrc file to manage this rather than setting the variables globally. The variables which we need to set are:

AWS_ACCESS_KEY - Your aws access key for the account
AWS_SECRET_ACCESS_KEY - The secret which corresponds to your access key
TF_VAR_SLACK_API_KEY - A valid API token which can read the users details in slack

file://terraform/lambda.tf

If we take a look at the terraform/lambda.tf we will see that we are defining a IAM role and policy for lambda which allows Lambda to write to the CloudWatch logs. We are also creating an aws_lambda_function resource.

resource "aws_lambda_function" "default" {
  s3_bucket        = "${aws_s3_bucket.default.id}"
  runtime          = "nodejs4.3"
  function_name    = "${var.lambda_function_name}"
  role             = "${aws_iam_role.iam_for_lambda.arn}"
  handler          = "index.handler"
  s3_key           = "${aws_s3_bucket_object.default.id}"
  source_code_hash = "${data.archive_file.lambdazip.output_md5}"

  environment = {
    variables = {
      SLACK_API_KEY = "${var.slack_api_key}"
    }
  }
}

s3_bucket is the s3 bucket where our application code is stored, and the value of this is a reference to another bit of Terraform config.
runtime is specifying that we would like to use the nodejs4.3 runtime for lambda.
function_name this is the name we would like to call our lambda function which is set in the file terraform/terraform.tfvars also, has a value of slackbot.
role this is the IAM role that we created earlier in the config.
s3_key this is the id of the zip file that we have uploaded to s3 which contains our source code. source_code_hash this is the hash code of our uploaded zip file, should we change the contents of the source code then this hash will be different and triggers a deployment of a new version.
environment for our application to function we need a slack API key, and for simplicity, we are adding this to the environment variables. It is important to note that it is possible to encrypt this using KMS however we are storing this plain text.

file://terraform/s3.tf

This file contains the terraform config to create an s3 bucket where our source code lives and also creates the archive and ensures that upon deploying, the archive is added to the bucket.

data "archive_file" "lambdazip" {
  type        = "zip"
  source_dir  = "${path.module}/../_build"
  output_path = "${path.module}/../lambda.zip"
}

resource "aws_s3_bucket" "default" {
  bucket = "${var.s3_bucket_name}"
  acl    = "private"

  tags {
    Name        = "${var.s3_bucket_name}"
    Environment = "Development"
  }
}

resource "aws_s3_bucket_object" "default" {
  bucket = "${aws_s3_bucket.default.bucket}"
  key    = "slackbot"
  source = "${path.module}/../lambda.zip"
  etag   = "${data.archive_file.lambdazip.output_md5}"
}

We can use a data element archive_file which will create a zip archive for us when terraform runs. We are setting the paths for the source_dir which is the folder created when we run make build, and the output is a file called lambda.zip in our source code root.
Terraform also has the capability to create an s3 bucket for us, if you already have a bucket that you would like to use then you can skip this resource element and directly add the arn to the bucket attribute of the aws_s3_bucket stanza. If already have an S3 bucket you would like to use for your source code remember that it must exist in the us-east-1 zone until Lex is rolled out across the other zones in AWS.

Deploying the function

To deploy the function we can run:

$ make package
$ make deploy

These commands install the yarn dependencies for testing and then run the test, build, and deploy steps, the deploy step simply runs terraform plan && terraform apply . The packaging step only needs to be run once or when you would like to update jest.

package:
  cd src && yarn install
test:
    cd src && yarn run jest

clean:
    rm -rf _build

build: clean
    mkdir _build
    cp src/index.js _build/
    cp src/dispatcher.js _build/
    cp src/slack.js _build/
    npm install --prefix=_build @slack/client

deploy: test build
    cd terraform && terraform plan
    cd terraform && terraform apply

You will see some output on the command line:

cd src && yarn run jest                                                                                                                                                                                                                                                                    yarn run v0.22.0                                                                                                                                                                                                                                                                           $ "/Users/nicj/Developer/lambda/slack-bot/src/node_modules/.bin/jest"                                                                                                                                                                                                                       PASS  ./dispatcher.test.js                                                                                                                                                                                                                                                                  dispatch                                                                                                                                                                                                                                                                                     ✓ calls the slack API with unknown user (4ms)                                                                                                                                                                                                                                              ✓ calls the slack API with known user (1ms)                                                                                                                                                                                                                                                ✓ returns error when error from slack API                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         Test Suites: 1 passed, 1 total                                                                                                                                                                                                                                                             Tests:       3 passed, 3 total                                                                                                                                                                                                                                                             Snapshots:   0 total                                                                                                                                                                                                                                                                       Time:        0.539s, estimated 1s                                                                                                                                                                                                                                                          Ran all test suites.                                                                                                                                                                                                                                                                         console.log dispatcher.js:88                                                                                                                                                                                                                                                                 dispatch userId=undefined, intent=WhereIs                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           console.log dispatcher.js:88
    dispatch userId=undefined, intent=WhereIs

  console.log dispatcher.js:88
    dispatch userId=undefined, intent=WhereIs

  console.log dispatcher.js:107
    error: Something went wrong
                                                                                                                                                                                                                                                                                           ✨  Done in 1.01s.

...

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path:

If you take a look at your AWS Lambda dashboard (N.Virginia) you should now see a new function and we can begin to configure Lex and Slack.

Step 3 Configuring AWS Lex and Slack

Sign in to the AWS Management Console and open the Amazon Lex console at https://console.aws.amazon.com/lex/.
Click the Create button and choose Custom Bot fill in the details and then press Create.
Click Create intent and the Create new Intent
Enter the title of the intent “WhatTimezone” and press Create
Add the utterances “What timezone is {User} in” and the slots User which has the type AMAZON.Person. Select our lambda function in the Fulfillment section and then save the intent.
To test our bot we need to build it so press the Build button in the top right-hand corner, the test window will then pop up and you can test your bot is working before connecting it to slack. If you have called your Intent something different to “WhatTimezone” then you need to update the code in src/dispatcher.js accordingly.
We can now publish the function by pressing the Publish button in the top right-hand corner and proceed to configure Slack.
The default instructions from AWS are just fine for configuring slack, so rather than reproduce them verbatim I encourage you to follow steps 3 through 6 in the following guide. http://docs.aws.amazon.com/lex/latest/dg/slack-bot-association.html. One important thing to note is that when you are setting up your channel you will be asked for a Verification Token, Slack only shows the Verification token on the Basic Information screen once you have added an Event Subscription however until you add the verification token AWS will not give you the URL. To get round this click on Event Subscriptions enter any URL and once you have a verification failed message again click on Basic Information.

Step 4 testing in Slack

Once everything has been configured, you can then go to your slack team invite the slackbot you have just created to the channel and then when you write a message like “What timezone is nicjackson in?”, the bot should reply with the correct answer.

Summary

This example is only a simple based on some investigation I did at the weekend into Lex; however, it has been quite fun, and I will certainly continue to work on this and add more advanced functionality. All the resources and links for this talk are listed below, and if you need any assistance, I can be found on Twitter.

Source Code:
https://github.com/nicholasjackson/slack-bot-lex-lambda
Lex Developer Guide:
http://docs.aws.amazon.com/lex/latest/dg/what-is.html
Lex Input and Response Format:
http://docs.aws.amazon.com/lex/latest/dg/lambda-input-response-format.html
Terraform:
https://www.terraform.io
Jest:
https://facebook.github.io/jest/

Configuring API Gateway for GraphQL Lambda functions

Tue, 13 Dec 2016 00:00:00 +0000

This is a short post which hopefully will explain a few gotchas when setting up a GraphQL endpoint with AWS Lambda.

The main problem is not Lambda per-se it is actually API Gateway which sits in front of Lambda. By default API Gateway expects input and output type of application/json and this is fine except when you send a POST request which contains a data payload in the format of application/graphql. The best practice document on the GraphQL website http://graphql.org/learn/serving-over-http/ recommends that a POST request should enable both requests in application/json format and in application/graphql.

If you try to send a query in graphql format to API Gateway like the following:

curl -i -XPOST -H 'Content-Type: application/graphql' -d '{ products { code } }' http://myendpoint.com/prod

You will find that you get an error message like the following:

HTTP/1.1 400 Bad Request
 Content-Type: application/json
 Content-Length: 189
 Connection: keep-alive
 Date: Tue, 13 Dec 2016 11:04:31 GMT
 x-amzn-RequestId: eca43820-c123-11e6-8b65-53d711979cac
 X-Cache: Error from cloudfront
 Via: 1.1 92423f766303b20998ffc97d623638f7.cloudfront.net (CloudFront)
 X-Amz-Cf-Id: MZVqwhSakST3qG4B_lKoGVuDdBvtwJfY4hkamF8pgDN-bb6S9e5ERg==
 
 {"message": "Could not parse request body into json: Unexpected character (\'p\' (code 112)): was expecting double-quote to start field name\n at [Source: [B@77acd1ac; line: 1, column: 4]"}%

This is due to API Gateway attempting to convert our post body into JSON, which it clearly is not.

How to fix this

To fix this problem we need to make some changes to the API Gateway configuration.

Step 1: Enable Binary Support

The first thing we need to do is to enable binary support, if you click the Binary Support menu item you will be presented with a screen which allows you to tell API Gateway to treat payloads associated with certain media types as binary data. This means that it will not attempt to convert the payload to JSON.

If we add our application/graphql media type and re-deploy the API you will see that we are now getting a slightly different error message.

HTTP/1.1 400 Bad Request
 Content-Type: application/json
 Content-Length: 203
 Connection: keep-alive
 Date: Tue, 13 Dec 2016 11:14:14 GMT
 x-amzn-RequestId: 47d5bff1-c125-11e6-aef0-459b4e1c48a4
 X-Cache: Error from cloudfront
 Via: 1.1 92423f766303b20998ffc97d623638f7.cloudfront.net (CloudFront)
 X-Amz-Cf-Id: xoaKYlLZg866CwA14f3p0u5N15sYxQcwiowCy1sdm9HDsmIFp0oxhw==
 
 {"message": "Could not parse request body into json: Unrecognized token \'eyBwcm9kdWN0cyB7IGNvZGUgfSB9\': was expecting (\'true\', \'false\' or \'null\')\n at [Source: [B@68df8622; line: 1, column: 57]"}%

Step 2: Add a mapping template

Just adding binary support is not enough, API Gateway is still attempting to parse the data into JSON however this is now happening in a later stage of the request, if you click on your resource and look at the Method Execution flow you should have an Integration Request item. Clicking on this will take you to the following screen and expanding the Body Mapping Templates section we need to add a new template.

Add a mapping template for the content type application/graphql and then configure your template by adding the following code.

{ "query": "$util.base64Decode($input.body)" }

This transforms the input into the output with the following structure.

{
  "query": "{ products { code title } }"
}

When we use the binary support our request body represented by $input.body is stored as a base64 encoded string. We can use the utility function base64Decode(string) to transform this back into a string. Lambda is still expecting a Payload in JSON format by using the mapping template we can take our input and wrap this into a format that the lambda function is happy with.

Now again deploy the API and re-test.

You should see your data now returning as requested.

HTTP/1.1 200 OK
 Content-Type: application/json
 Content-Length: 1622
 Connection: keep-alive
 Date: Tue, 13 Dec 2016 11:34:07 GMT
 x-amzn-RequestId: 0c943239-c128-11e6-94b6-7967c2010156
 X-Amzn-Trace-Id: Root=1-584fdcab-b22f22b3fe1fc4d5323d71c1
 X-Cache: Miss from cloudfront
 Via: 1.1 be27f8aa8af4ef7f3d1f074ecbf3b266.cloudfront.net (CloudFront)
 X-Amz-Cf-Id: NDfYQ70rUlqMrJt-_gjRMBCYDEj5eED7JM8bPCkFMRb7ZGXoWgmiFQ==
 
 {"data":{"products":[{"code":176176},{"code":465956},{"code":277748}]}}%

One thing to note is that your lambda function must be returning the content type as application/json if it is not you might find that instead of returning JSON API Gateway wraps your output in a string which will most likely break things.

Building microservices with Swift : Part 1 of 3

Wed, 07 Dec 2016 00:00:00 +0000

POST COMING SOON

What makes a good microservice framework

Sun, 21 Aug 2016 00:00:00 +0000

This is of course an entirely subjective opinion for most people however whilst I have been thinking about this and how it could be possible to score and rank the various frameworks I thought I would share a mind map I have made.

Ease of use?

The first category I wanted to look at was the Ease of use, any good framework should be easy to use, this category has been broken down into the following sub categories.

Fast setup

The framework should be fast to setup and with the minimum number of steps and dependencies.

Code generation

It should support code generation templates to scaffold an initial service and possibly extend the service.

Tooling

There must be adequate tooling.

CI/CD: Must integrate and work well with Continuous Integration and Continuous Deployment pipelines. The tooling must be scriptable from bash.
Cross platform: The tools must work cross platform, Mac and Linux as a bare minimum.

Maintainable

The framework must be maintainable with the minimum of effort.

Easy to update: It must be easy to update with the minimum of code changes.
Well versioned: The framework must be well versioned with breaking changes to the API mainly restricted to Major version updates.

Format

Ideally the service will support good standards in one of the following formats.

REST

If the framework implements REST it must take full advantage of semantic API design with appropriate use of HTTP verbs and Status Codes.

RPC

If the framework is RPC based it must use a standard and open messaging protocol.

Patterns

The framework must implement the standard patterns of microservice architecture.

Circuit breaking

Client calls to downstream services must implement circuit breaking.

Service discovery

It must be capable of registering with a dynamic service registry and capable of querying the same registry to locate connected services.

Consul
Etcd
DNS SRV records
Proprietory: Proprietary service registries must be open and usable from other clients which do no implement the framework or its SDKs.

Timeouts

Downstream client calls must be equipped with a user configurable timeout.

Health checks

The framework must create an automatic health check endpoint.

Routing

The framework must support multiple routes with an easy to use pattern based matching.

Middleware

The framework must support middleware to allow the user to create shared code for handlers.

Load balancing

Downstream client connections should be capable of load balancing.

Language independence

The framework needs to be language independent to enable cross team polyglot workflow.

Polyglot

At a minimum it should be possible to create client SDKs in multiple languages.

Ability to interface with other frameworks

It must be possible to interact with any service built with the framework by clients who are not built using the same framework.

Implements standards

A standard message protocol should be used to maximize interaction.

JSON-RPC
Thrift
Protobuffers
JSON

Open

The framework should be open in both the source code and the roadmap.

Extensible

When required the user should be able to extend the framework.

Plugins

A pluggable software architecture to be able to create generators and templates.

Middleware

Extension through handler middleware.

Support

A good support network is incredibly important throughout the lifecycle of the service.

Maintained

The framework should be well maintained with:

Regular updates
Accepts pull requests
Corporate sponsor: Whilst this option is not essential a corporate sponsor can extend the lifecycle of a framework as there is less likelihood of a [leftpad] situation.

Documentation

The framework should be well documented with clear and concise examples and comprehensive API documentation.

Easy to follow: Documentation should be accessible and easy to read.
Code samples: Adequate code samples should be provided to support a developer using the framework.

Tutorials

The framework will ideally have community contributed tutorials in both blog and video formats.

Community

There should be a good community using and supporting the framework with at least one of the following channels of communication.

Slack
Gitter
Github
Mailing list
Stack Overflow

Secure

The framework should be secure and implement the latest industry standards.

TLS

It should be possible to secure the endpoints of the framework using TLS.

OWASP

The framework should implement OWASP advisory.

Validation

Requests should be automatically validated based on rules implemented by message annotation.

Well patched

Security vulnerabilities should be regularly assessed and patched.

Open source

The framework should be open sourced and released on a license which allows forking and modification.

Community

There should be good open source community following and contribution for the project.

Quality

The code quality of the service should be visible and of a high standard. Community contributions should follow a published process and standard.

High test coverage

Test coverage should be high and monitored, pull requests should ensure adherence to coding standards.

Unit High number of fast running unit tests is essential for both the framework and the generated code.
Behavioural / functional: Ideally the framework should implement behavioral and functional tests in terms of the generated code and the build process.

Automated builds

Automated builds of the source code should be present and visible. Pull requests should run an automated build and the state reported on the request.

Code quality

Automated code quality tools should be used and the results visible.

Coveralls
Code climate
Sonar

Standard language patterns

A standard method of writing the code taking account of the language level idioms is essential.

Efficient

The framework must produce code which is efficient when run.

Fast

The code must execute quickly and be designed for performance.

Low latency

Requests should be low latency.

Low memory

The service should be memory efficient.

Supports large number of connections

It should support a large number of concurrent connections.

My struggles with RPC

Fri, 19 Aug 2016 00:00:00 +0000

I have decided to write up my thinking on some investigation I have done with gRPC and protobuffers. At first I hated the concept as I was fully entrenched in the “Why use anything other than REST” camp, however from playing round a little with gRPC I am starting to feel that RPC with a DSL for defining contracts with protobuffers could be the way to go.

My argument for REST has always been the accessibility of it, however gRPC supports 10 languages from the official packages and many more from community packages. The benefits of protobuffers are that you can generate a client in your chosen language using the protoc application, this allows DRYness of code across the many consumers of your API. Sure you can use Swagger to define your API and then code gen from the Swagger files but this process defeats the purpose of defining an API in a RESTful way will all the semantics of HTTP verbs to infer intent. If you are generating clients then why not take advantage of the efficiency of RPC based communication and also allow yourself the benefits of using light weight binary messaging.

Versioning:

No more complex than when working with REST, use semantic versioning can be implemented in the same way as you can in rest, gRPC is not like JMI where the exact same interface must be implemented on both the client and the server, you can add elements to the server side interface and by implementing the same standard rules you would follow with a RESTful API you are in a no worse position.

Breaking changes (update Major version, client contract update required):

Removing or renaming APIs or API parameters;
Changing the type of an API parameter for example from Integer to String;
Changes to response codes, error codes or fault contracts;
Changes to the behavior of an existing API.

Things that do not involve a breaking change (No client updated required unless desired):

Adding parameters to a returned entity;
Adding additional endpoints or functionality;
Bug fixes or other maintenance which do not included items in the breaking changes list.

Web access:

This is where RPC falls down however there is an architectural pattern called a HTML → RPC reverse proxy which is explained a little bit in this blog post: (http://www.grpc.io/blog/coreos)[http://www.grpc.io/blog/coreos]. Whilst this can seem like a problem it is restricted to your Public APIs. Following the reverse proxy pattern you can easily create RESTful endpoints for browser based clients and support RPC for native apps. Internal services can consume the RPC client with no problems.

Documentation

Your proto files are the documentation, creating comments inside the file is all that is really needed,

// The greeting service definition.
service Users {
  // Create user creates a user in the system with the given User details,
  // it returns an Error message which will be nil on a successful operation
  rpc CreateUser (User) returns (Error) {}
}

// Person describes a user entity
message User {
  // name is a required field and represents the name of
  required string name = 1;
  // id is the unique identifier for the user in the system
  required int32 id = 2;
  // email is the users email address and is an optional field 
  optional string email = 3;
}

message Error {
  optional code int32 = 1
  optional detail string = 2
}

Advantages / Disadvantages

Advantages

Ability to easily implement one / two way streaming for data;
Auto generated client code;
Fast and low latency communication;
Native mobile clients get a huge boost by avoiding HTTP;
Fantastic open source support with gRPC, Thrift, JSON-RPC.

Disadvantages

Inability to leverage HTTP caching provided by a CDN;
Possibility of having to implement a reverse proxy to support browser based clients;
Testing, you can’t just CURL an endpoint and it may be required to build a test harness.

Conclusion

I will certainly be investigating gRPC and protobuffers further and can see myself using this as the defacto communication standard for my microservices.

You call that optimisation? This is optimisation!

Sat, 23 Jul 2016 00:00:00 +0000

Whilst writing a chapter for my book “Building Microservices in Go” I was working on a section for JSON parsing in Go. As I was explaining the various semantics of the encoding/json package I wrote a really simple example of using json.Marshal to convert a struct into it most simple string representation. In this example (see listing 1) I decoded the struct into a byte array and then wrote it to the response using fmt.FPrint. Not the code I would write in a handler but to explain the simple useage of the package before moving on to show how you can use the prefered method which are Encoders (see listing 2).

listing 1

func BenchmarkHelloHandlerVariable(b *testing.B) {                                                                                
        b.ResetTimer()                                                                                                            
                                                                                                                                  
        var writer = ioutil.Discard                                                                                               
        response := Response{Message: "Hello World"}                                                                              
                                                                                                                                  
        for i := 0; i < b.N; i++ {                                                                                                
                data, _ := json.Marshal(response)                                                                                 
                fmt.Fprint(writer, string(data))                                                                                  
        }                                                                                                                         
}                                                                                                                                 
                                                                                                                                  

listing 2

func BenchmarkHelloHandlerEncoder(b *testing.B) {                                                                                 
        b.ResetTimer()                                                                                                            
                                                                                                                                  
        var writer = ioutil.Discard                                                                                               
        response := Response{Message: "Hello World"}                                                                              
                                                                                                                                  
        for i := 0; i < b.N; i++ {                                                                                                
                encoder := json.NewEncoder(writer)                                                                                
                encoder.Encode(response)                                                                                         
        }                                                                                                                         
}          

After writing this code I decided that I would have a quick look and see the inefficiencies between the first method and the second. It turns out there is a 40% speed improvement in the second example. This is mainly due to the fact that the using the encoder.Encode rather than copying the output from Marshal to a byte array dramatically reduces the number of memory allocations that the method has to perform from 5 to 2.

If we optimise this further by passing a reference to the encoder rather than passing by value (see listing 3) we see further gains by 100 nanoseconds a 50% reduction over the first method, again this is because we are reducing the number of memory allocations performed.

listing 3

func BenchmarkHelloHandlerEncoderReference(b *testing.B) {                                                                                 
        b.ResetTimer()                                                                                                            
                                                                                                                                  
        var writer = ioutil.Discard                                                                                               
        response := Response{Message: "Hello World"}                                                                              
                                                                                                                                  
        for i := 0; i < b.N; i++ {                                                                                                
                encoder := json.NewEncoder(writer)                                                                                
                encoder.Encode(&response)                                                                                         
        }                                                                                                                         
}          

benchmarks

go test -v -run="none" -bench=. -benchtime="5s"  -benchmem
testing: warning: no tests to run
PASS
BenchmarkHelloHandlerVariable           10000000              1187 ns/op             248 B/op          5 allocs/op
BenchmarkHelloHandlerEncoder            10000000               731 ns/op              24 B/op          2 allocs/op
BenchmarkHelloHandlerEncoderReference   10000000               660 ns/op               8 B/op          1 allocs/op
ok      github.com/nicholasjackson/building-microservices-in-go/chapter1/bench  28.394s

Now these are tiny numbers but if this was the only operation that the server was performing then we could get away with half the hardware to satisfy the same load, that means saving money.

I hope this has shown you that allocating memory does not come for free and that with a little thought and knowledge you can dramatically increase the speed of your application. I would never advocate premature optimisation and there is none of this here, this is simply understanding that memory allocation takes time and if we can avoid it then we should. It is also about using the right options that you already have available to you and to do that you need to understand your core language and framework.

0 to Microservice in 5 minutes with Go and Minke

Fri, 26 Feb 2016 00:00:00 +0000

Introduction

I went to a conference a few weeks ago which was angled at technology in the retail environment, by lunchtime nobody had mentioned the words “Docker” or “Microservices”, I made my excuses and got the hell out of there. Everyone is talking about Microservices, even the bus driver the other day was complaining that the reason the bus was late was due to not enough monitoring in their scheduling microservice. Allegedly the wrong config was pushed to Consul, the service was failing and nobody knew. OK so the last part might be made up but I bet a few of you will have felt that pain, those who have not, don’t worry, when it happens it will be OK. Unless of course you work on nuclear missile launch systems and you accidental wipe out life for half the planet, that would not be OK, very not OK.

The thing is that everyone is talking about microservices but the frameworks and development practices are not so mature yet, there are some amazing companies like Netflix and USwitch who have been using this stuff for a number of years but knowledge and patterns have only slowly seeped out into the mainstream. Rails was not built in a day and it will be the same for a good microservice framework, there is some amazing stuff on the horizon but the key will be mass adoption and contribution.

What I would like to show you in this post is how you can go from 0 - Microservice in 5 minutes, that’s scaffolding a new service building and testing locally then getting the build and test passing on CI. I will assume that you have the Docker Toolkit, Ruby and Go installed on your dev machine as otherwise that might take longer than 5 minutes and that would mess with the title of my post.

Frameworks

I started to build my own framework when I was re-developing a project originally written in C#, I wanted something which was minimal and would cover 90% of my use cases, something which worked with my rather opinionated method of development and testing which is Google’s Go as my language, Cucumber and Ruby for my functional test suite, Docker stuff for my platform.

When I first started to build microservices in Go I found that I would repeat the same scaffolding steps time and time again. I would copy some boiler plate code for my functional tests and also for my build scripts. It all became a little unmanageable by the time I had finished the second service, I had also learned a lot. Thats when I first built go-microservice-template this worked really well for a while but I found that whilst it was fine to add the new learnings to the template it was a total pain to update existing services.

Time for split, go-microservice-template was put out to pasture and the build and test framework was split out into a separate Rubygem called Minke. With Minke I implemented the concept of Generators which would support scaffolding and building applications in multiple languages not just Go. So far this has worked well, localising my build and test code into a gem has recently proven useful when I switched from my own Jenkins server to CircleCI and found myself unable to use the Docker exec command. Being able to update a central gem saved hours and hours of copy and pasting.

Go Microservice Generator for Minke

Installing and scaffolding a microservice

Assuming your GOPATH is set correctly to get a copy of the template on your local computer and you have Ruby 2.x you need two gems installed:

$ gem install minke
$ gem install minke-generator-go

This will install the two gems into your gem cache, to generate your service you simply execute following command (obviously changing the paths unless your github user is nicholasjackson in which case you will be me).

$ minke -g minke-generator-go -o $GOPATH/src/github.com/nicholasjackson/hello -n github.com/nicholasjackson -a hello

The options for the above command are:

flag
-g	generator template to use
-o	output folder for generated code
-n	namespace for application
-a	application name

The application will then scaffold the service for you and place the contents at:

$GOPATH/src/[your_namespace]/[service_name]

The service should be ready to build and deploy, complete with a couple of example endpoints, unit and functional tests.

Sprint 0

Scaffold the service and get it deploying to production. If I had pound for every time a project has been delayed due to the complexities (software and people) of getting a service deployed to production I would currently be surfing and not writing this post. I designed the template to be running and testable straight out of the box so that the next task can be to get it running in the continuous delivery pipeline.

I recommend that this is not only set up for Dev and Test environments but you build the pipeline into production. You do not need to expose the service but I certainly recommend deploying to production every time you commit to master as a matter of good practice.

A closer look at the scaffolded service?

As I mentioned earlier I am a fan of the Bauhaus approach to my scaffolding; clean lines and minimalist. There are however a few key things that I find I can not do without in every service.

Dependency Injection - My current favorite is the Facebook inject package github.com/facebookgo/inject.
Routing - Tried and tested it has to be Gorilla all the way github.com/gorilla
Request Validation - As my friend Yan Ettles would say “Input Validation, Input Validation, Input Validation”, security is essential and this is a good place to start github.com/asaskevich/govalidator.
Logging - Metrics are incredibly important, personally I like to use StatsD for simple metrics you can store these in a service like Datadog or if you need to host your own you can use Graphite and Carbon. For more detailed logs Syslogd will do the trick you can export your logs to either something like Loggly or your own ELK (Elasticsearch, Logstash, Kibana) stack.

Dependency Injection

The Facebook framework is excellent allowing you to inject:

Shared instances of objects.
Named instances of objects, useful when the framework can not infer from the type such as when using an interface.
Private objects which are unique to each injected class.

This does not feel very Go like and there are a lot of question on why you need to use dependency injection in Go when you can just have a shared package which stores references. For me it is largely about the ugliness and brittleness of the code when using a shared package but also Facebook inject has some nice features like it will traverse a tree when creating an object, automatically creating and injecting sub dependencies. Take a look at handlers/health.go and handlers/health_test.go for an example of use, the initial graph creation takes place in the main startup method.

There is some weird code in the health handler which I must warn you about:

type HealthResponseBuilder struct {
  statusMessage string
}

func (b *HealthResponseBuilder) SetStatusMessage(message string) *HealthResponseBuilder {
  b.statusMessage = message
  return b
}

func (b *HealthResponseBuilder) Build() HealthResponse {
  var hr HealthResponse
  hr.StatusMessage = b.statusMessage
  return hr
}

type HealthDependenciesContainer struct {
  // if not specified will create singleton
  SingletonBuilder *HealthResponseBuilder `inject:""`

  // statsD interface must use a name type as injection cannot infer ducktypes
  Stats logging.StatsD                    `inject:"statsd"`

  // if not specified in the graph will automatically create private instance
  PrivateBuilder *HealthResponseBuilder  `inject:"private"`
}

I generally would not write code like this for such a simple purpose it is purely as an example to myself as to what is possible with Facebook inject.

Routing

When it comes to routing I have used the Gorilla Toolkit since my first go application, it is fast and does everything I need. The handlers package deals with the routing and setting up a new handler with Gorilla is a breeze.

r.Get("/v1/health", HealthHandler)

r.Add("POST", "/v1/echo", requestValidationHandler(
  ECHO_HANDLER+POST,
  reflect.TypeOf(Echo{}),
  RouterDependencies.StatsD,
  http.HandlerFunc(EchoHandler),
))

We are using two different ways of registering a handler the simpler Get which takes a path and function and the more complex Add. The more complex option is so we can add some middleware before our handler gets called, this is where we perform our validation and what I will get on to next.

Request Validation

The go validator package github.com/asaskevich/govalidator is awesome, it allows us to use struct annotation to define what valid is for our properties and then by simply calling a method on the validator package we get an array of error messages and an error object if the validation fails.

type Echo struct {
  Echo string `json:"echo" valid:"stringlength(1|255),required"`
}

request = Echo{}
request.Echo = "Valid String"

errors, err = govalidator.ValidateStruct(request)

When we wire up this code into middleware like in the requestValidationHandler we have the ability to ensure our routing goes through the middleware before the handler, this means we can implement painless request validation for every handler. govalidator has just about every possible scenario you could like to cover including some really nice things like validating a string as a url or net address all is explained in the package docs.

Logging

You need logging and I recommend lots and lots of metrics out via StatsD or something similar. There are various options from Gauges, Counters, Timing Summary Statistics, and Sets but whatever you use having the right data will not only help you debug and diagnose a problem but you can setup alerting in systems such as Datadog to wake you up in the middle of the night when something goes wrong, and believe me it will go wrong. The simple implementation in the template is just using Increment (Counter) but it is easy to add other measures if required. When you run the application Graphite is also spun up which is accessible at http://192.168.99.100:8080/ where 192.168.99.100 is the ip address of your docker machine.

I have also stated an opinion on a way to define the tags for the various metrics in const.go in the handlers package. Ideally you do not want to only create a common format for metric tags in your microservice but treat it like a coding standard. Stick to the same format for every service, it will make dashboard creation a thousand times easier and you will not have to spend a week refactoring all the tags across 20 microservices because they were difficult to understand.

package handlers

const GET = ".get"
const POST = ".post"
const CALLED = ".called"
const SUCCESS = ".success"
const BAD_REQUEST = ".bad_request"
const INVALID_REQUEST = ".invalid_request"
const VALID_REQUEST = ".valid_request"

const HEALTH_HANDLER = "helloworld.health_handler"
const ECHO_HANDLER = "helloworld.echo_handler"

Testing

Testing with go-microservice-template is broken into two groups:

Unit tests (Go)
Functional tests (Cucumber)

I follow an outside in methodology when writing code, this is my preference it works for me therefore my framework does this too. I did say this was an opinionated framework didn’t I?

Unit tests

The nice thing about Go is that you can test handlers independently from the HTTP server which allows you to tests the logic of your handler really quickly. Combine this with the dependency injection framework to isolate dependencies or inject mocks, you have a really efficient and fast way to do test driven development.

The below code is one of the tests, we are mocking the request and checking the response, since there is no HTTP server involved these tests are actually really fast to run. This gives us the ability to write full coverage on our handlers as well as other logic, the only bit not covered is the wiring of the router and the start of the service but this will get coverage from the functional tests.

func TestEchoHandlerCorrectlyEchosResponse(t *testing.T) {
  echoTestSetup(t)

  var responseRecorder *httptest.ResponseRecorder
  var request http.Request

  responseRecorder = httptest.NewRecorder()

  echo := Echo{Echo: "Hello World"}
  context.Set(&request, "request", &echo)

  EchoHandler(responseRecorder, &request)

  body := responseRecorder.Body.Bytes()
  response := Echo{}
  json.Unmarshal(body, &response)

  assert.Equal(t, 200, responseRecorder.Code)
  assert.Equal(t, response.Echo, "Hello World")
}

Functional tests

All the functional tests are in the _build/features folder and are written in Gherkin. Cucumber is my weapon of choice, Ruby and Rspec are just perfect for the job and there are quite a few cucumber plugins which have been developed meaning you need to write very few actual step implementations. Following this method you gain the ability to create some oh so essential integration tests and a really nice way to express the behavior of the service which defines the contract between you and the client.

Scenario: Echo returns same data as posted
  Given I send a POST request to "/v1/echo" with the following
    | echo | Hello World |
  Then the response status should be "200"
  And the JSON response should have "$..echo" with the text "Hello World"

By using a gem such as cucumber-rest-api the above cucumber feature will execute without needing a single step file (NOTE: version 0.4 of this gem is broken). That is not so difficult to maintain, all you cucumber haters complaining about your slow and brittle test suite already screaming NOOOO. I’m going to make a statement, you won’t like it and it will most likely fill my comments with hate but here goes anyway.

It’s not cucumber it’s you!

Test the behavior of your contract, a little integration and trust your unit tests to cover the rest, more often than not test coverage in functional tests is way too high. The other benefit of microservices are that your bloated test coverage will be distributed across multiple repos, instant test paralyzation.

Minke

Minke is an opinionated build system for Docker based microservices like go-microservice-template it encapsulates all the horrible boilerplate build scripts and wraps them up so you only need to deal with config files like the extract below. I will not go to much into depth with minke it is worth reading the readme at https://nicholasjackson.github.com/minke however we will look at the commands which build and test your microservice.

---
application_name: 'helloworld'
namespace: 'github.com/nicholasjackson'
generator_name: 'minke-generator-go'
docker_registry:
  url: <%= ENV['DOCKER_REGISTRY_URL'] %>
  user: <%= ENV['DOCKER_REGISTRY_USER'] %>
  password: <%= ENV['DOCKER_REGISTRY_PASS'] %>
  email: <%= ENV['DOCKER_REGISTRY_EMAIL'] %>
  namespace: <%= ENV['DOCKER_NAMESPACE'] %>

Inside the _build folder is a Gemfile with all the Ruby dependencies in it like Minke, before we can build our service we need to install them, you need a Ruby interpreter, personally I recommend RVM to manage the different versions of Ruby and all the dependencies.

Ensure Docker server is running and you have the docker environment variables set.
Install the gems with bundle install.
Build the service rake app:build_image this will get the go package dependencies, run the unit tests, build the application for linux then wrap everything nicely up into a Docker image.
Run the functional tests rake app:cucumber to spin up our mock environment using Docker Compose and execute the Cucumber tests against it. All things well you would see the following output from the terminal, if something did not work it will most likely be because you have not got your dependencies like Docker installed or set up correctly.

Consul

To feed config to our service we will use consul and consul template, the consul template can be found in _build/dockerfile/[servicename]/config.ctmpl. Consul template runs as a service in the container and checks for changes in Consul server. It will write the value that corresponds to the key and restart the server when a change is detected. The config itself is stored in a yaml file consul_keys.yml which Minke loads into consul on the start of the tests or when you run the application. Below is a simple example of a Consul template for our service, Consul is written in Go and the templates are actually Go templates.

{
  "stats_d_server": "{ {key "app/stats_d_server"} }"
}

Docker Container

Ah Docker, the thing pay rises are made from, I love Docker, in fact I loath to use anything else for packaging and running an application these days. With the rise of services like Docker Cloud, Amazon Container Service and Google Container Service, long gone are the days of fighting with Chef and all things Unix which I have never fully understood. Docker is also just about perfect, no essential if you are building a microservice architecture, the pain it causes will be an order of magnitude lower than not using it.

Minke builds your code and wraps it up in a nice warm Docker blankie, check out the _build/dockerfile/[servicename]/ for more details but the dockerfile that go-microservice-template creates is based on Alpine linux which creates a nice small image, you can read more about that part of the process in another post here: http://nicholasjackson.github.io/other/micro-docker-images-for-microservices/.

Docker Compose

When you run or test your application Minke uses Docker Compose, if you examine the compose file in the build folder, you will see entries for consul and our StatsD server. As my service grows in complexity so does this file, in order to control behavior from dependencies when I am testing my code I generally use mock servers like Mimic. I try keep real dependencies like databases or queues to a minimum, as a rule of thumb on how micro is micro I would suggest that if you need to connect to more than one datastore your service might be two big. All other connections should be to other services which can be mocked.

version: '2'
services:
  helloworld_test:
    image: helloworld
    ports:
      - "::8001"
    environment:
      - "CONSUL=consul:8500"
    links:
      - consul:consul
      - statsd:statsd
  consul:
    image: progrium/consul
    ports:
      - "::8500"
    hostname: node1
    command: "-server -bootstrap -ui-dir /ui"
  statsd:
    image: hopsoft/graphite-statsd
    ports:
      - "::80"
    expose:
      - "8125/udp"
    environment:
      - "SERVICE_NAME=statsd"
  registrator:
    image: 'gliderlabs/registrator:latest'
    links:
      - consul:consul
    command: '-internal -tags=dev consul://consul:8500'
    volumes:
      - '/var/run/docker.sock:/tmp/docker.sock'

CI/CD

Sprint 0 get helloworld running under ci and deploying to an environment, I mentioned this earlier but I can not stress how much it will save your ass to keep your CI pipeline healthy, I have recently switched to CircleCI as they are capable of building both iOS and Android as well as my microservices. To get a Go based microservice running on CircleCI took a little wrangling at first as whilst the auto-detection and build process is nice it does not allow for Minke to do its magic. It is simple enough to configure to run the way we want as the below config shows.

To get your service building all you need to do is change the paths in the dependencies section to point to your namespace and project name. I have much love for CircleCI right now my builds take about 4 minutes for a service and most of that is setting up the environment, the UI is nice and clean and easy to use and the ability to SSH into the build host was absolutely essential when I was trying to figure out this config.

CircleCI Config

machine:
  pre:
    - curl -sSL https://s3.amazonaws.com/circle-downloads/install-circleci-docker.sh | bash -s -- 1.10.0
    - sudo rm /usr/local/bin/docker-compose
    - sudo curl -L https://github.com/docker/compose/releases/download/1.7.1/docker-compose-`uname -s`-`uname -m` > docker-compose
    - sudo mv docker-compose /usr/local/bin/
    - chmod +x /usr/local/bin/docker-compose
  ruby:
    version: 2.3.1
  services:
    - docker
  environment:
    GOPATH: /home/ubuntu/go

dependencies:
  override:
    - cd _build && bundle && bundle update
    - mkdir -p /home/ubuntu/go/src/github.com/nicholasjackson
    - cp -R /home/ubuntu/helloworld /home/ubuntu/go/src/github.com/nicholasjackson/

test:
  override:
    - cd /home/ubuntu/go/src/github.com/nicholasjackson/helloworld/_build && rake app:build_image && rake app:cucumber

Conclusion

Thats it we are done, 0 to microservice in 5 minutes, following the simple steps:

Scaffold a new service (go run generate.go)
Run the build script (rake app:build_image)
Run the tests (rake app:cucumber)
Create the circle.yml config
Tell CircleCI to watch your project
Push the code to your repo
Boil the kettle and sit back and wait for the build to go green

For me this will always be work in progress and as I learn more I will update both Minke and minke-generator-go but for now this pattern is really working for me. If you have any comments please feel free to ping me a message, find a bug, always happy for contributions.

In the next post I am going to write more in-depth about service discovery and some software patterns for managing disconnected services.

Source Code

If you need it please check out the example project at https://github.com/nicholasjackson/helloworld.

Dependencies

Stuff to install or nothing will work and you will swear at me.

Ruby https://rvm.io/
Go https://golang.org/
Docker Toolbox including Docker Compose https://www.docker.com/products/docker-toolbox
CircleCI https://circleci.com/