Core v2: A Java EE application

Since quite some time my home automation is build around a core app made in Java EE. As part of this challenge I’m improving the core and integrating it more tightly through MQTT. The core is responsible for coordinating all nodes and enforcing the business rules for our home.

Introduction

Thuis ArchitectureThe Thuis core runs on the Raspberry Pi 3 in a WildFly container. Like all nodes it’s connected to the MQTT broker for communication with the rest of the system. It also takes care of communication with some external applications, like Plex (media server) and Netatmo (weather station). In this blog I’ll focus on the integration of MQTT, Z-Way devices and the rules.

The building blocks of the application are as follows:

  • Model:
    • Devices
    • Rooms
    • Scenes
    • Rules
  • Commands
  • Controller
  • MQTT-CDI

Each of these will be described in this blog. Currently all models and rules are defined as static objects in Java code, the goal is to put this in the database at some point and make it editable through a UI, that will however be after the challenge ends.

Device Model

The two base models are Device and Room. To be combined with Device there are different types of Actuator and Sensor. The following image and table show the interfaces. There are several implementations, for example there is MqttSwitch which implements Device and Switch and defines a switch which can be controlled through MQTT.

Device, Actuator and Sensor class diagram

[table width=”650″ colwidth=”100|400″]
Interface, Description
Device, General definition of a (virtual) device: an identifier and it’s status (type will be overridden by more specific interfaces)
Actuator, A device with controls
Switch~~extends Actuator, A switch with on/off and toggle features
Dimmer~~extends Switch, A switch which also has can have values between 0 and 100\, where 0 is off
Thermostat~~extends Switch, A thermostat supporting a set point for the temperature. It can be turned on/off\, which will put it on predefined on and off temperatures
Sensor, A device that provides a (single) value of a sensor
BinarySensor~~extends Sensor, A sensor with a value that can be true or false
MultilevelSensor~~extends Sensor, A sensor with an arbitrary value and a configurable unit of measurement
[/table]

All devices are virtual devices, each with a single function. For example a sensor that can measure both temperature and trigger on movement will be implemented as two sensors: a MultilevelSensor and a BinarySensor.

In this blog post we’ll describe two of our rooms: the living room and the kitchen. This gives us the following definitions:

package nl.edubits.thuis.server.devices;

public class Rooms {
	public static Room living = new Room("living");
	public static Room office = new Room("office");
	public static Room kitchen = new Room("kitchen");
}
package nl.edubits.thuis.server.devices;

import static nl.edubits.thuis.server.devices.Rooms.living;
import static nl.edubits.thuis.server.devices.Rooms.kitchen;
/* other imports */

public class Devices {
	public static MqttSwitch livingMoodTop = new MqttSwitch(living, "moodTop");
	public static MqttSwitch livingMoodBottom = new MqttSwitch(living, "moodBottom");
	public static MqttSwitch livingMoodChristmas = new MqttSwitch(living, "moodChristmas");
	public static MqttDimmer livingMain = new MqttDimmer(living, "main");

	public static MqttBinarySensor kitchenMovement = new MqttBinarySensor(kitchen, "movement");
	public static MqttMultiLevelSensor kitchenTemperature = new MqttMultiLevelSensor(kitchen, "temperature", Units.CELSIUS);
	public static MqttMultiLevelSensor kitchenIlluminance = new MqttMultiLevelSensor(kitchen, "illuminance", Units.LUX);
	public static MqttSwitch kitchenMicrowave = new MqttSwitch(kitchen, "microwave");
	public static MqttSwitch kitchenCounter = new MqttSwitch(kitchen, "counter");
	public static MqttDimmer kitchenMain = new MqttDimmer(kitchen, "main");
}

Command Model

As you might have noticed in the interfaces above, several methods return Command. A command is a runnable class that can be executed to fulfill a task, for example turning on a Switch with Switch.on(). For each type of command there is an implementation. The one used most by the devices defined above is the MqttCommand which publishes a MQTT message, so for example Z-Way will receive it and take action. The implementation is quite straight forward:

package nl.edubits.thuis.server.automation.commands;

/* imports */

public class MqttCommand implements Command {
	String topic;
	String content;

	public MqttCommand(String topic, String content) {
		this.topic = topic;
		this.content = content;
	}

	@Override
	public void runSingle() {
		MqttService mqttService = CDI.current().select(MqttService.class).get();
		mqttService.publishMessage(topic, content);
	}
}

Command class diagram

Commands can be encapsulated in other commands. By encapsulating you can either compose several commands into a single one, or add a condition to the command. The following commands of this type exist:

[table width=”650″ colwidth=”100|400″]
Command, Description
ConditionalCommand, A command that executes the encapsulated command when a certain condition is met. The condition is defined using a Predicate lambda. Three default conditions are available:~~
illuminance – shortcut for predicates based on illuminance\, for example execute when illuminance is below 100lux~~
whenOn/whenOff – execute a command when a given device is turned on/off
ListCommand, execute several commands in order
PrioritizedCommand, execute a command with a different priority\, for example USER_INITIATED: the highest priority which will let the command jump in front of the execution queue
WaitForCommand, wait for a condition to be true before executing another command. Two default conditions are available: waitForOn and waitForOff which wait for the status of a device to turn on or off before executing another command
[/table]

Another way of combining commands is using a Scene. This is an object that contains two ListCommand, one for activating the scene and one for deactivating it. A scene for turning on and off the mood lighting in the living room is defined like:

package nl.edubits.thuis.server.devices;

import static nl.edubits.thuis.server.devices.Devices.livingMoodBottom;
import static nl.edubits.thuis.server.devices.Devices.livingMoodChristmas;
import static nl.edubits.thuis.server.devices.Devices.livingMoodTop;
/* other imports */

public class Scenes {
	public static Scene mood = new Scene("mood",
		asList(
			livingMoodTop.on(),
			livingMoodBottom.on(),
			livingMoodChristmas.on()
		),
		asList(
			livingMoodTop.off(),
			livingMoodBottom.off(),
			livingMoodChristmas.off()
		)
	);
}

Observing MQTT messages

The Core observes MQTT messages arriving on basically any topic. It then checks if there are any devices (or better ObserveMqttStatus implementations) matching this topic. The status of these devices is then updated. When a sensor gets a new value an event is emitted. These events (and individual MQTT messages) can triggered rules. This all happens in the MqttObserverBean. This bean also takes care of updating the status of any scenes or rooms including this device.

The connection with the MQTT broker is handled by the MQTT-CDI extension made by Alexis Hassler, to which I contributed some improvements in the past. This CDI extension abstracts the actual connection away. When MQTT messages arrive on a subscribed topic they are fired as CDI events which can be observed using the @MqttTopic annotation. This way you can very easily observe any messages arriving:

public void onMessageLivingMain(@Observes @MqttTopic("Thuis/device/living/main") MqttMessage message) {
	logger.log("Light in the living was turned "+message.asText());
}

For publishing messages a service method is available.

Rules

To enable sensors (or other events) to trigger commands there are rules. A rule is an Observer of either an MQTT topic or a SensorChanged event. As result one or more commands are executed. An example of a rule is the following:

package nl.edubits.thuis.server.automation;

/* imports */

@ApplicationScoped
public class Rules {

	@Inject
	private Controller controller;

	public void onKitchenMovement(@Observes @SensorChange("kitchen/movement") BinarySensor sensor) {
		LocalTime now = LocalTime.now();

		if (sensor.getStatus() && Devices.kitchenIlluminance.isLowerOrEqual(80)) {
			if (TimeUtils.isBetween(now, LocalTime.of(6, 0), LocalTime.of(10, 0))
					|| TimeUtils.isBetween(now, LocalTime.of(12, 30), LocalTime.of(13, 30))
					|| TimeUtils.isBetween(now, LocalTime.of(20, 30), LocalTime.of(21, 30))) {
				// Breakfast/Lunch/After dinner
				controller.run(userInitiatedPriority(Devices.kitchenMicrowave.on()));
				controller.run(userInitiatedPriority(Devices.kitchenCounter.on()));
			} else if (TimeUtils.isBetween(now, LocalTime.of(17, 30), LocalTime.of(20, 30))) {
				// Dinner
				controller.run(userInitiatedPriority(Devices.kitchenMicrowave.on()));
				controller.run(userInitiatedPriority(Devices.kitchenCounter.on()));
				controller.run(userInitiatedPriority(Devices.kitchenMain.on()));
			} else {
				controller.run(userInitiatedPriority(Devices.kitchenMicrowave.on()));
			}
		} else {
			controller.run(Devices.kitchenMicrowave.off());
			controller.run(Devices.kitchenCounter.off());
			controller.run(Devices.kitchenMain.off());
		}
	}
}

This example covers most of the basic options. It observes the movement sensor in the kitchen. When it’s status becomes true and it’s not very light it will check 3 timeframes. Depending in which timeframe the current time fits a combination of lights is turned on. This way you always have the most useful lights for the task ahead. The timing might need some optimization, but this is a good start.

Controller

Taking care of the actual execution of commands and scenes are the Controller and the CommandExecutor. Take the example rule above: it triggers several lights to be turned on or off. These commands are passed on to the controller. The controller takes the command, determines it’s priority and then puts it on a JMS queue:

package nl.edubits.thuis.server.controller;

/* imports */

@Stateless
public class Controller {

	@Inject
	private JMSContext context;

	@Resource(mappedName = Resources.COMMAND_QUEUE)
	private Queue commandQueue;

	public void run(Command command) {
		run(command, 0);
	}

	public void run(Command command, long deliveryDelay) {
		context.createProducer()
		       .setPriority(command.getPriority().getValue())
		       .setDeliveryDelay(deliveryDelay)
		       .send(commandQueue, command);
	}
}

(for the purpose of this blogpost the code is simplified a bit)

A JMS MessageListener, the CommandExecutor, is used to listen to the commands which are added to the queue and execute them. Because of the way the JMS queue works, commands are executed in chronological order while respecting the priorities. This means that all commands with the same priority are executed exactly in order they were added (FIFO), but when a command of a higher priority is added to the queue it’s moved in front. This is used for situations where for example the home theater is starting up (which takes a few minutes in total) and someone triggers a motion sensor. This command gets the USER_INITIATED priority and is therefore executed at the first possible moment, in front of all other steps of starting up the home theater. Something that’s not time sensitive (for example automatically turning off the heating at night) gets a LOW priority and will therefore never block any more important commands.

Some commands can take a long time and you don’t want them to block the queue. For example a WaitForCommand takes until the condition becomes true. In this case the condition is tested once and when the result isn’t true yet the command is added to the queue again with a small timeout.

Concluding

The most important part of the Core is now done, but most rules still have to be implemented. In a later stage some more external systems will be added to the core, for example for controlling the Home Theatre. I’m also aware that, to keep this blog post from growing too much, I have simplified some code samples and didn’t cover every detail. If you’re interested in a certain detail, please let me know and I’ll explain it more!

Publishing activity from Z-Way to MQTT

Now we have a working Raspberry Pi 3 with Z-Way it’s time to connect it to the backbone of Thuis: MQTT. For this I developed a custom userModule/app. In this post I’ll explain what it does, how to use it and how it works.

Goal

The app enables Z-Way to send and receive MQTT messages. Whenever the status of one of the selected devices changes it will be published to a topic. Based on these topics some other topics are available to change the status of the devices or request a status update.

How to use

Install

Z-Way MQTTZ-Way allows developers to create userModules and publish them to their App Store. Unfortunately they still didn’t reply to my submit, so it’s available straight away. Luckily they do have a beta-token mechanism, so it’s still possible to install it. I’ve used BaseModule by @maros as basis, so you’ll need to install this as well.

To install it follow these steps:

  1. Go to the Management page of your Z-Way
  2. Open the App Store Access tab
  3. Add mqtt_beta as token
  4. Now go to Apps, and then Online Apps
  5. Search for Base Module and install it
  6. Search for MQTT and install it

Configure

MQTT PublicationWhen you go to the settings of the app you’ll find quite some options. The first section are the basic settings needed for MQTT: the client ID and hostname/port/username/password of the MQTT broker. Next are the common topic prefix (we’ll use Thuis/device for now) and postfixes for requesting a status update and for setting the status of a device.

More interesting is the second section, which are the publications. When you add a publication you first select a type: a single device or tagged devices. We’ll use the latter for now. Now you will add some tags to select which devices will be part of this publication.

Next is deciding on the topic for the publication. You can use two placeholders: %deviceName% and %roomName%. They will be replaced by the actual values for the corresponding device. In this example we’ll use the topic %roomName%/%deviceName%.

The last option is wether or not to publish status updates as retained messages, we’ll turn it on.

Receive status updates

Now you’ve configured the Z-Way MQTT app so we can use it to receive updates on our devices. We’ll use mosquitto_sub to demonstrate this:

robin@thuis-server-core:~# mosquitto_sub -v -t Thuis/#
Thuis/device/kitchen/counterSwitch on
Thuis/device/kitchen/counterSwitch off
Thuis/device/office/dimmer 0
Thuis/device/kitchen/luminescence 39

As you can see a devices called ‘Counter Switch’ in the room ‘Kitchen’ was turned on and off, the dimmer in the office was turned down, and the luminescence sensor in the kitchen gave an updated status.

Interact

There are two interactions available: requesting a status update and setting a value. Both can be demonstrated using mosquitto_pub. To request the status of the dimmer in the office publish a message to Thuis/device/office/dimmer/status with an empty message:

robin@thuis-server-core:~# mosquitto_pub -t Thuis/device/office/dimmer/status -m ""

When you subscribe to the dimmer’s topic in the meanwhile you receive the new status:

robin@thuis-server-core:~# mosquitto_sub -v -t Thuis/device/office/dimmer
Thuis/device/office/dimmer 0

Then to set the dimmer to 60% you send the message 60 to Thuis/device/office/dimmer/set:

robin@thuis-server-core:~# mosquitto_pub -t Thuis/device/office/dimmer/set -m "60"

The light will turn on and you’ll see the status changed:

robin@thuis-server-core:~# mosquitto_sub -v -t Thuis/device/office/dimmer
Thuis/device/office/dimmer 60

Currently values between 0 and 100 are supported for dimmers (devices supporting the SwitchMultiLevel command class) and on/off for switches (devices supporting the SwitchBinary command class).

How does it work

Z-Way apps are made in JavaScript. Their basic structure is defined in the developer manual. As said before I’m using the BaseModule as basis as it provides some useful functions for interacting with devices, for example filtering status updates to includes only updates with actual changes. The full project can be found on my GitHub as Zway-MQTT. Here I’ll explain some interesting parts of the code.

MQTT client

From a userModule you can call out to external utilities, but only when you give explicit permission. I would like to avoid this, so I searched for a pure JavaScript solution. Most JavaScript-based MQTT libraries use WebSockets, which is unfortunately not available in Z-Way. Luckily I found a module by @goodfield. He found and modified a MQTT client for use within Z-Way. I cleaned it up and started using it:

self.client = new MQTTClient(self.config.host, parseInt(self.config.port), {client_id: self.config.clientId});

Publishing status updates

To receive updates we have to subscribe to updates sent by the Z-Way core. I’m using the modify:metrics:level event, which is the filtered version from BaseModule.

self.callback = _.bind(self.updateDevice, self);
self.controller.devices.on("modify:metrics:level", self.callback);

In updateDevice I retrieve the new value from the device and transform it if needed. Then I’ll look up all publications matching this device (for example because they are tagged with the configured tag) and execute a MQTT publish for them.

MQTT.prototype.updateDevice = function (device) {
	var self = this;

	var value = device.get("metrics:level");
	if (device.get("deviceType") == "switchBinary" || device.get("deviceType") == "sensorBinary") {
		if (value == 0) {
			value = "off";
		} else if (value == 255) {
			value = "on";
		}
	}

	self.processPublicationsForDevice(device, function (device, publication) {
		var topic = self.createTopic(publication.topic, device);
		self.publish(topic, value, publication.retained);
	});
};

You’ll notice the createTopic call, this call takes care of merging the prefix and configured topics plus it replaces the placeholders. Device and room names are camel cased to have nice and valid topics.

MQTT.prototype.createTopic = function (pattern, device) {
	var self = this;

	var topicParts = [].concat(self.config.topicPrefix.split("/"))
		.concat(pattern.split("/"));

	if (device != undefined) {
		topicParts = topicParts.map(function (part) {
			return part.replace("%roomName%", self.findRoom(device.get("location")).title.toCamelCase())
					   .replace("%deviceName%", device.get("metrics:title").toCamelCase());
		});
	}

	return topicParts.filter(function (part) {
		return part !== undefined && part.length > 0;
	}).join("/");
};

Reacting on interaction

To react on status requests and new values the app will subscribe to all topics starting with the prefix. It will then filter out the actions. If it receives an action message it will try to find the corresponding publication and device bases on the topic. It will then take the requested action.

self.client.subscribe(self.createTopic("/#"), {}, function (topic, payload) {
	var topic = topic.toString();

	if (!topic.endsWith(self.config.topicPostfixStatus) && !topic.endsWith(self.config.topicPostfixSet)) {
		return;
	}

	self.controller.devices.each(function (device) {
		self.processPublicationsForDevice(device, function (device, publication) {
			var deviceTopic = self.createTopic(publication.topic, device);

			if (topic == deviceTopic + "/" + self.config.topicPostfixStatus) {
				self.updateDevice(device);
			}

			if (topic == deviceTopic + "/" + self.config.topicPostfixSet) {
				var deviceType = device.get('deviceType');

				if (deviceType.startsWith("sensor")) {
					self.error("Can't perform action on sensor " + device.get("metrics:title"));
					return;
				}

				if (deviceType === "switchMultilevel" && payload !== "on" && payload !== "off") {
					device.performCommand("exact", {level: payload + "%"});
				} else {
					device.performCommand(payload);
				}
			}
		});
	});
});

We can now easily talk to Z-Wave devices through MQTT messages. Later I’ll add support for more command classes, like thermostat and power usage. Someone already requested battery level updates through GitHub, which is an interesting addition as well.

Installing the Raspberry Pi 3 in practice

Last week I explained how I’m using Chef to provision my Raspberry Pi’s and the recipes in my Thuis cookbook. Back then I didn’t have a Raspberry Pi 3 yet, so I tested it on an older model. This week the kit arrived, so I’ll bring the bootstrapping in practice on the Raspberry Pi 3! Not everything went as expected, so this is a good subject for this blogpost.

The Kit

Pi IoT Kit

Installing the Raspberry Pi 3

As mentioned before I’m using raspbian-ua-netinst as a basis for my install, as this gives me a very lean install of Raspbian. The maintainers of the projects didn’t update the installer yet for the Raspberry Pi 3, so there are some manual steps to go through for now. This is discussed in issue #375. It boils down to the following steps:

  1. Format the SD card
  2. Copy all files from the latest version to the SD card
  3. Fix the files for the Raspberry Pi 3:
    1. Copy (replace all) the files from the firmware repo using the handy tarball made by @eLvErDe
    2. Edit /boot/config.txt by adding the following at the end:
      [pi3]
      initramfs installer-rpi2.cpio.gz
  4. Insert the SD card in the Raspberry Pi and power it on
  5. Now wait until the installation finishes (it will take quite some time, when you don’t have a screen attached wait until the ethernet connection becomes silent)
  6. SSH into the Raspberry using the default username root and password raspbian
  7. Now fix the kernel by installing and running rpi-update:
    apt-get install rpi-update
    rpi-update
    reboot now
    
  8. Edit /boot/config.txt again by changing [pi2] into [pi3]

The Raspberry Pi is now ready to be bootstrapped.

Bootstrapping Chef

Bootstrapping Chef is a matter of running one simple command from my workstation:

knife bootstrap 10.0.0.201 -t raspbian-jessie-chef.erb --ssh-user root --ssh-password 'raspbian' --node-name 'thuis-server-core' --run-list 'recipe[thuis::thuis-server-core]'

However I ran into a few complications. During my testing I didn’t run into these because of the iterative process and some differences between the hardware of the two Raspberries.

Complication 1: systemd needs services to be reloaded

After installing WildFly the recipe tries to start the services; this however fails with a cryptic No such file or directory. The reason for this happens to be that systemd needs to be reloaded after adding new init-scripts. The WildFly recipe doesn’t take care of this and therefor runs into the issue. To resolve the issue I forked the cookbook and added a systemctl daemon-reload command to it (see PR #38). This ensures it can start WildFly as expected.

Complication 2: For Z-Way bluetooth needs to be turned off

RazberryThe RazBerry uses the serial IO pins on the Raspberry Pi, the same are used by the built-in Bluetooth connection as well. To be able to use the RazBerry and run Z-Way, Bluetooth has to be disabled. The original install script takes care of this, but after I converted it to be used in a Chef cookbook this didn’t work anymore. That’s why I converted these commands to Ruby blocks in my Chef recipe now. This looks as follows:

ruby_block '/etc/inittab' do
	block do
		file = Chef::Util::FileEdit.new('/etc/inittab')
		file.search_file_delete_line(/[^:]*:[^:]*:respawn:\/sbin\/getty[^:]*ttyAMA0[^:]*/)
		file.write_file
	end
end

ruby_block '/boot/cmdline.txt' do
	block do
		file = Chef::Util::FileEdit.new('/boot/cmdline.txt')
		file.search_file_delete(/console=ttyAMA0,115200/)
		file.search_file_delete(/kgdboc=ttyAMA0,115200/)
		file.search_file_delete(/console=serial0,115200/)
		file.write_file
	end
end

ruby_block '/boot/config.txt' do
	block do
		file = Chef::Util::FileEdit.new('/boot/config.txt')
		file.insert_line_if_no_match(/dtoverlay=pi3-miniuart-bt/, 'dtoverlay=pi3-miniuart-bt')
		file.write_file
	end
end

Basically this deletes the serial connections from /etc/inittab and /boot/cmdline.txt and adds an option to /boot/config.txt to disable Bluetooth.

Complication 3: sudo commands not found

When trying to do a command using sudo, e.g. sudo shutdown now I got a Command not found error. This happens because /sbin and other secure directories are not on the PATH for a non-root user. When you use sudo it should be and for this a small piece of configuration is added for the sudo cookbook:

default['authorization']['sudo']['sudoers_defaults'] = [
		'env_reset',
		'secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"'
]

Configuring Z-Way

As I already have a working Z-Way setup at home the configuration is pretty easy. It boils down to:

  1. Browse to http://10.0.0.201:8083 (where 10.0.0.201 is the IP of your Raspberry)
  2. Choose a password of your own choice
  3. Restore the backups:
    1. For the controller (*.zbk): http://10.0.0.201:8083/expert/#/network/control
    2. And for the module automation (*.zab): http://10.0.0.201:8083/smarthome/#/admin
    3. If you used userModules then copy or install these again (their settings are part of the backup, but not the actual files)

Deploying an application to WildFly

This is easy as dropping an .ear of .war file to /opt/wildfly/standalone/deployments. Alternatively you can deploy the app as part of a Chef recipe like:

wildfly_deploy 'my-app-1.0.war' do
  url 'http://artifacts.company.com/artifacts/my-app.1.0.war'
  runtime_name 'my-app.war'
end

In the upcoming blogs I’ll work on the Java app and integrating Z-Way with MQTT, then we’ll do the actual deployment.

With the complications out of the way, the Raspberry Pi 3 is now fully ready to receive some app deployments, which is what I’ll work on now!

 

Cooking up the nodes: Thuis Cookbook

In my last post I explained the basics of Chef, and in the last week I worked on defining the configuration of each node. I selected several cookbooks from the Supermarket and wrote some myself. Using a series of recipes I defined the software and configuration of two of the nodes of Thuis in the Thuis Cookbook. In this post I’ll show you my choices and give some code samples to let you set up your own Chef config.

Let’s start with some of the cookbooks I’m using from the Supermarket:

[table width=”600″ colwidth=”25%|75%” colalign=”right|left|left”]
Cookbook,Description
apt, Takes care of keeping the apt-get cache up to date
firewall, Install and configure UFW
hostnames, Automatically configures the hostname of each node based on a pattern
mosquitto, Install and configure Mosquitto (MQTT broker)
tar, Download and extract a tar file
sshd, Install and configure the SSH deamon
sudo, Configure which users can sudo
timezone_lwrp, Configure the time zone
users, Add default users with SSH keys
wildfly, Install and configure WildFly and Java
[/table]

Basics: default recipe

Execute on each the device the default recipe provides the basic setup of a node. All it does is installing a few packages and including other recipes:

package ['rpi-update', 'nano']

include_recipe 'apt'
include_recipe 'hostnames'
include_recipe 'timezone_lwrp'
include_recipe 'mosquitto::client'
include_recipe 'thuis::firewall'
include_recipe 'thuis::sshd'
include_recipe 'thuis::users'

Of course the recipes have some configuration, this is provided through the attributes file:

# System
default['set_fqdn'] = '*.internal.thuisapp.com'
default['tz'] = 'Europe/Amsterdam'
default['authorization']['sudo']['passwordless'] = true

# Firewall
default['firewall']['allow_ssh'] = true
default['thuis']['open_ports'] = [80, 8080, 9990] # TODO: finetune

The firewall, sshd and users recipes are part of the thuis cookbook, the first two are fairly straight forward:

# Include the default firewall recipe
include_recipe 'firewall::default'

# Allow incoming on the configures ports
ports = node['thuis']['open_ports']
firewall_rule "open ports #{ports}" do
  port ports
end
# Don't allow logins through SSH using a password
openssh_server node['sshd']['config_file'] do
	PasswordAuthentication 'no'
end

The users recipe is based on this nice blogpost: it’s creating my default user (robin), adding my SSH key and allows it to sudo without using a password.

Specifics per node

For both nodes (more will follow later) I’ve created a specific recipe. These recipes include the default recipe plus take care of the specific needs of that node.

thuis-server-core

As mentioned in my post about the architecture of Thuis the core node will use WildFly, a MQTT broker and Z-Way:

include_recipe 'thuis::default'

include_recipe 'mosquitto'
include_recipe 'thuis::wildfly'
include_recipe 'z-way'

Mosquitto for now uses the default configure of the cookbook, I’ll finetune this later. I did have to update the cookbook a bit as it didn’t have support for Debian/Raspbian Jessie yet, for that I did a PR, which is accepted.

WildFly required a bit more effort as the standalone.xml configuration file wasn’t update with the latest version and the underlying java cookbook doesn’t have support for the ARM packages needed for the Raspberry Pi. The latter was solvable using some custom configuration which selects a different download based on the architecture:

# Java
default['java']['arch'] = kernel['machine'] =~ /x86_64/ ? 'x86_64' : kernel['machine'] =~ /armv/ ? 'armhf' : 'i586'

java_home_arch = 'i386'
if (node['kernel']['machine'] == 'x86_64')
	java_home_arch = 'amd64'
end
if (node['kernel']['machine'] =~ /armv/)
	java_home_arch = 'armhf'
end
force_default['java']['java_home'] = "/usr/lib/jvm/java-#{node['java']['jdk_version']}-#{node['java']['install_flavor']}-#{java_home_arch}"

default['java']['jdk']['8']['armhf']['url'] = 'http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-arm32-vfp-hflt.tar.gz'
default['java']['jdk']['8']['armhf']['checksum'] = '79dda1dec6ccd7130b5204e75d1a8300e5b02c18f70888697f51764a777e5339'

default['java']['jdk']['8']['x86_64']['url'] = 'http://download.oracle.com/otn-pub/java/jdk/8u91-b14/jdk-8u91-linux-x64.tar.gz'
default['java']['jdk']['8']['x86_64']['checksum'] = '6f9b516addfc22907787896517e400a62f35e0de4a7b4d864b26b61dbe1b7552'

Next is overriding the WildFly standalone.xml configuration:

include_recipe 'wildfly::default'

resources("template[#{::File.join(node['wildfly']['base'], 'standalone', 'configuration', node['wildfly']['sa']['conf'])}]").cookbook 'thuis'

The template file is a copy of the original, but updated using a diff between the original and the version from WildFly 10.0.0.Final. The needed configuration in attributes.rb is:

# WildFly
default['wildfly']['version'] = '10.0.0.Final'
default['wildfly']['url'] = 'http://download.jboss.org/wildfly/10.0.0.Final/wildfly-10.0.0.Final.tar.gz'
default['wildfly']['checksum'] = 'e00c4e4852add7ac09693e7600c91be40fa5f2791d0b232e768c00b2cb20a84b'
default['wildfly']['enforce_config'] = true
default['wildfly']['mysql']['enabled'] = false
default['wildfly']['postgresql']['enabled'] = false
default['wildfly']['jpda']['enabled'] = false
default['wildfly']['java_opts']['other'] = ['-client']

It changes the version to use, disables a few modules and important on the Raspberry Pi it uses -client instead of -server in Java options.

Chef Z-Way cookbook directory structureThe most difficult to get working was Z-Way. There is no cookbook available yet, so I had to build this one from scratch. I could have taken a relative easy way out by just letting the cookbook execute Z-Way’s install script, but as I want to learn Chef I went for the hard road. On the right you can see the structure of the cookbook. The full cookbook is available on Github at Edubits/chef-z-way.

The cookbook uses a few recipes to install Z-Way to the device, and more importantly to safely upgrade it while retaining the configuration of all devices. It also installs the required services and enables them to automatically start like:

service 'z-way-server' do
	service_name 'z-way-server'
	supports restart: true
	action [:enable, :start]
end

A few things are kept as in the original script in the template file install.sh.erb and executed during as part of the install recipe. As I’m currently testing on an old Raspberry Pi with the Razberry hardware, I could not verify the full installation yet.

thuis-server-tv

The TV node will mostly take care of connecting to the home cinema system using CEC. For this a Java EE application is used, so WildFly is needed here as well. Next to WildFly we need libcec.

include_recipe 'thuis::default'

include_recipe 'thuis::wildfly'
include_recipe 'thuis::libcec'

libcec in Jessie is only version 2 and we need 3, so we’ll grab this package from the Stretch repository:

# Add stretch apt repository
apt_repository 'stretch' do
	uri          'http://archive.raspbian.org/raspbian/'
	distribution 'stretch'
	components   ['main']
end

package ['libcec3', 'cec-utils'] do
	default_release 'stretch'
end

Testing

Raspberry Pi 1BChef has a very nice way of testing cookbooks using Kitchen and Vagrant: it spins up a virtual machine from an image and runs the recipes on that. I used this for the general testing, however quite some of my changes and configuration are specifically made for the Raspberry Pi. This means to test those I had to use an actual device. For this I used my good old Raspberry Pi 1B. This required some patience, as one chef-client run without any changes takes about 5 minutes on this device. As there are some differences in architecture between the 1B and the 3B I expect there will be some small changes needed when deploying the cookbooks to the new device.

Bootstrap the nodes

As soon as the kit arrives (somewhere in the coming week) I will install them using raspbian-ua-netinst, assign a static IP in my router and then I can bootstrap the Raspberry Pi’s with just one command each:

knife bootstrap 10.0.0.201 -t raspbian-jessie-chef.erb --ssh-user root --ssh-password 'raspbian' --node-name 'thuis-server-core' --run-list 'recipe[thuis::thuis-server-core]'
knife bootstrap 10.0.0.202 -t raspbian-jessie-chef.erb --ssh-user root --ssh-password 'raspbian' --node-name 'thuis-server-tv' --run-list 'recipe[thuis::thuis-server-tv]'

Now the Pi’s are fully installed and ready to be used and it’s time to actually build & deploy automation software and connect some hardware! Let the fun begin 🙂

Using Chef to provision Raspberry Pi’s

ChefInspired by one of the other challengers I started to experiment with Chef. Frederick is using Puppet to provision his Raspberry Pi’s. To get more knowledge on the several tools out there we decided to both try one. So that’s why I started using Chef.

Chef vs Puppet

Both tools are open source projects built for automatically provisioning nodes with software and configuration. They both have a fairly similar setup using a server and clients on all nodes. The biggest difference is in how you manage your configuration. Puppet uses a Ruby-based DSL which is similar to JSON, Chef uses pure Ruby. This makes Chef a bit more powerful out-of-the-box. As Rich Morrow said in his report:

Whereas Chef tries to provide more power to the user, Puppet puts safety rails around them.

Some noteworthy differences:

[table width=”500″ colwidth=”20%|40%|40%” colalign=”right|left|left”]
,Chef,Puppet
Language,Ruby, DSL
Execution, Order enforced, Model driven
Approach, Programmer’s approach, Sysadmin friendly
Used by a.o., Facebook & Adobe, Twitter & Intel
[/table]

To learn more about Puppet I refer you to Frederick’s posts, here more about Chef!

The basics

A typical setup consists of three elements: your workstation, a server, and nodes.

Chef setup

The server is the central repository for all code and it also keeps knowledge about every node it manages. From your workstation you write and verify the configuration policy and then upload it to the server. When you run chef-client on a node the latest code is downloaded from the server and the node’s configuration is brought up-to-date.

Preparation

To start you need a set up workstation and server. You can install a server locally or use a hosted version. Hosted Chef is free up to 5 nodes, so that’s how we’ll start.

  1. Install the Chef Development Kit
  2. Sign up for the trial of Hosted Chef
  3. Create an Organization at https://manage.chef.io/
  4. Go to the Administration tab, selection your new organization and click Generate Knife Config
  5. Save knife.rb to ~/.chef
  6. Copy your private key (created during signup) to ~/.chef as well
  7. Test the connection between your workstation and the server with knife ssl check

Your first cookbook & recipe

A cookbook is a set of configuration describing a service or node. It consists of recipes, template files and attributes. A recipe describes everything that is required to configure part of a system, for example which software packages to install, how to configure them or execute other recipes.

For this example we’ll create a simple cookbook which creates a file, just like in Fredericks example. Start with generating the cookbook:

chef generate cookbook test-chef

You now have the following directory structure:

.
└── test-chef
    ├── Berksfile
    ├── chefignore
    ├── metadata.rb
    ├── README.md
    ├── recipes
    │   └── default.rb
    ├── spec
    │   ├── spec_helper.rb
    │   └── unit
    │       └── recipes
    │           └── default_spec.rb
    └── test
        └── integration
            ├── default
            │   └── serverspec
            │       └── default_spec.rb
            └── helpers
                └── serverspec
                    └── spec_helper.rb

As you can see there is already a default recipe created. Let’s edit that to contain the following:

file '/tmp/testfile' do
  content 'test content'
  mode '0444'
end

Now upload it to your server:

knife cookbook upload test-chef

And bootstrap a node:

knife bootstrap ADDRESS --ssh-user USER --ssh-password 'PASSWORD' --sudo --use-sudo-password --node-name node1 --run-list 'recipe[test-chef]'

You’ll see that Chef is installed on the node and in the end the file is created according to the recipe. When you run chef-client from the node it will check if there is a new version and if the file is still there. If something is not according to the recipe – it will fix it.

Supermarket

Of course there are already lots of cookbooks created by others, they are shared in the Supermarket. You can easily add a dependency to one of these cookbooks. There are several ways to override attributes used in the cookbook, or even override complete files to make sure it does exactly what you want.

You should have a basic understanding of Chef now, there is however much more. To let you get more familiar with Chef they have some excellent tutorials. In upcoming blogposts I’ll show my cookbooks as well.

The ugly: Chef on a Raspberry Pi

Now the bad news: Chef doesn’t support the Raspberry Pi out-of-the-box, so you can’t bootstrap it as easily as on other platforms. It also needs a fairly recent version of Ruby, which is not available in the Jessie repository. Luckily you can overcome this by using a custom bootstrap script and using the Stretch (the upcoming version of Debian/Raspbian) repository.

A good starting point is the Raspbian-Bootstrap by Dayne. It’s made for Wheezy, so we’ll have to update it a bit. It took some experimenting, but at some point I found a working solution. We can skip the custom Ruby build (which takes a long time) by using the ruby2.3 package from the Stretch repository and I’ve updated the syntax of the configuration part to the one from the default Chef bootstrap.

This brings us to the following steps:

  • Take a Raspberry Pi with a clean Raspbian install (I use raspbian-ua-netinst, as it gives a minimal install)
  • Download my version of Raspbian-Bootstrap
  • Use knife to bootstrap with a custom template:
    knife bootstrap PI_IP_ADDRESS -t raspbian-jessie-chef.erb --ssh-user root --ssh-password 'raspbian' --node-name 'NODE_NAME' --run-list 'recipe[thuis-base::default]'

    This will do the following:

    • Add the Raspbian Stretch repository to Apt and update the Apt index
    • Remove any existing versions of Ruby
    • Install Ruby 2.3 plus build tools from the Stretch repository
    • Install gems needed for Chef + Chef itself
    • Add basic configuration for Chef
    • Start chef-client for the first time running in this case my base cookbook thuis-base
  • Start using Chef on your Raspberry!