rise of the machines - part 2

Here I show how to use C++ to communicate via Bluetooth with the LEGO Mindstorms EV3 brick (see previous post).

If you are on a Mac everything should work right away. If you are using Ubuntu or other Linux distro I think you’ll only need to change the Bluetooth part a bit (my Ubuntu laptop doesn’t have Bluetooth, so I can’t be sure). If somehow you are forced to use Windows I think you’ll need to change the Bluetooth part a lot. All the rest should be the same though.

So, you start by cloning the source code of the EV3 firmware: open up your terminal and do git clone https://github.com/mindboards/ev3sources.git Name the folder ev3sources, to make the examples below easier to run. Also, open the ev3sources/lms2012/c_com/source/c_com.h file and change the line #include "lms2012.h" in order to provide the full path to the lms2012.h file. Say:

#include "/Users/YourUsername/MyLegoProject/ev3sources/lms2012/lms2012/source/lms2012.h"

That’s all the setup you need - you are now ready to write and send commands to the EV3. Turn on your EV3, enable Bluetooth, make it discoverable (see the EV3 user guide if necessary), plug some motor to port A, fire up Xcode or whatever IDE you use, and try running the following code snippet:

#include <unistd.h>
#include <fcntl.h>
#include "ev3sources/lms2012/c_com/source/c_com.h"

int main()
{
    
    // write command to start motor on port A with power 20
    unsigned const char start_motor[] {13, 0, 0, 0,
        DIRECT_COMMAND_NO_REPLY,
        0, 0,
        opOUTPUT_POWER, LC0(0), LC0(1), LC1(20),
        opOUTPUT_START, LC0(0), LC0(1)};

    // send command to EV3 via Bluetooth
    int bt = open("/dev/tty.EV3-SerialPort", O_RDWR);
    write(bt, start_motor, 15);

    // end connection with EV3
    close(bt);
}

If everything went well you should see the motor starting.

If instead you get an authentication-related error message, download and install the official LEGO app (if you haven’t already), launch it, use it to connect to the EV3 via Bluetooth, check that it really connected, then close it. Somehow that fixes the issue for good. (I know, it’s an ugly hack, but life is short).

Now let’s deconstruct our little script. There are two steps: writing the command and sending the command. Writing the command is the hard part. As you see, it’s not as simple as, say, EV3.start_motor(port = "A", power = 20). Instead of human-readable code what we have here is something called bytecodes. In this particular example every comma-separated piece of the expression inside the inner curly braces is a bytecode - except for the LC1(20) part, which is two bytecodes (more on this in a moment). The first and second bytecodes - 13 and 0 - tell the EV3 the message size (not counting the 13 and the 0 themselves). The third and fourth bytecodes - 0 and 0 - are the message counter.

The fifth bytecode - DIRECT_COMMAND_NO_REPLY - tells the EV3 two things. First, that the instruction is a direct command, as opposed to a system command. Direct commands let you interact with the EV3 and the motors and sensors. System commands let you do things like write to files, create directories, and update the firmware. Second, DIRECT_COMMAND_NO_REPLY tells the EV3 that this is a one-way communication: just start the motor, no need to send any data back. So, the three alternatives to DIRECT_COMMAND_NO_REPLY are SYSTEM_COMMAND_NO_REPLY, DIRECT_COMMAND_REPLY, and SYSTEM_COMMAND_REPLY.

The sixth and seventh bytecodes - 0 and 0 - are, respectively, the number of global and local variables you will need when receiving data from the EV3. Here we’re using a DIRECT_COMMAND_NO_REPLY type of command, so there is no response from the EV3 and hence both bytecodes are zero.

Now we get to the command lui-même. We actually have two commands here, one after the other. The first one, opOUTPUT_POWER, sets how much power to send to the motor. The second one, opOUTPUT_START, starts the motor. Each command is followed by a bunch of local constants (that’s what LC stands for), which contain the necessary arguments. For both commands the first LC0() is zero unless you have multiple EV3 bricks (you can join up to four EV3 bricks together; that’s called a “daisy chain”). Also for both commands, the second LC0() determines the EV3 port. Here we’re using port A - hence LC0(1). Use LC0(2) for port B, LC0(4) for port C, and LC0(8) for port D. Finally, opOUTPUT_POWER takes one additional argument: the desired power. The unit here is percentages: 20 means that we want the motor to run at 20% of its maximum capacity. Unlike the other local constants, this one is of type LC1, not LC0, so it takes up two bytes (see the bytecodes.h file for more on local constants); that is why the message size is 13 even though we only have 12 comma-separated elements.

(Don’t be a sloppy coder like me: instead of having these magic numbers, declare proper variables or constants and use these instead - LC0(port), LC1(power), etc.)

Now let’s send the command we just wrote. On a Mac the way we communicate with other devices via Bluetooth is by writing to (and reading from) tty files that live in the \dev folder (these are not actual files, but file-like objects). If you inspect that folder you will see one tty file for every Bluetooth device you have paired with your computer: your cell phone, your printer, etc. The EV3 file is called tty.EV3-SerialPort. (If you’re curious, here’s all the specs and intricacies of how Bluetooth is implemented on a Mac.)

So, to send the command we wrote before to the EV3 via Bluetooth we open the tty.EV3-SerialPort file (line 16), write the command to it (line 17), and close it (line 20).

That’s it, you can now use C++ to control the EV3 motors.

Just so you know, your command is automatically converted to hexadecimal format before being sent to the EV3 (those LC()s are macros that make the conversion). In other words, your EV3 will not receive {13, 0, 0, 0, DIRECT_COMMAND_NO_REPLY, 0, 0, opOUTPUT_POWER, LC0(0), LC0(1), LC1(20), opOUTPUT_START, LC0(0), LC0(1)}. It will receive \x0D\x00\x00\x00\x80\x00\x00\xA4\x00\x01\x81\x14\xA6\x00\x01 instead. The mapping is provided in the bytecodes.h file. For instance, DIRECT_COMMAND_NO_REPLY is 0x80, opOUTPUT_POWER is 0xA4, and so on.

If you prefer you can hardcode the hexadecimals. This produces the exact same outcome:

#include <unistd.h>
#include <fcntl.h>
#include "ev3sources/lms2012/c_com/source/c_com.h"

int main()
{
    
    // write command to start motor on port A with power 20
    char start_motor[] = "\x0D\x00\x00\x00\x80\x00\x00\xA4\x00\x01\x81\x14\xA6\x00\x01";

    // send command to EV3 via Bluetooth
    int bt = open("/dev/tty.EV3-SerialPort", O_RDWR);
    write(bt, start_motor, 15);

    // end connection with EV3
    close(bt);
}

If you master the hexadecimals you can use any language to communicate with the EV3. For instance, in Python you can do this:

# write command to start motor on port A with power 20
start_motor = '\x0D\x00\x00\x00\x80\x00\x00\xA4\x00\x01\x81\x14\xA6\x00\x01' + '\n'

# send command to EV3 via Bluetooth
with open('/dev/tty.EV3-SerialPort, mode = 'w+', buffering = 0) as bt:
    bt.write(start_motor)

All right then. Now, how do we get data back from the EV3? Well, it’s the reverse process: instead of writing to tty.EV3-SerialPort we read from it. The trick here is to find the sensor data amidst all the other stuff that the EV3 sends back to your computer, but we’ll get there (btw, I’m grateful to the good samaritan who showed me how to do this). To make matters more clear, plug some sensor on port 1 and try running this code:

#include <unistd.h>
#include <fcntl.h>
#include <iostream>
#include "ev3sources/lms2012/c_com/source/c_com.h"

int main()
{
    
    // read sensor on port 1
    unsigned const char read_sensor[] {11, 0, 0, 0,
        DIRECT_COMMAND_REPLY,
        1, 0,
        opINPUT_READ, LC0(0), LC0(0), LC0(0), LC0(0), GV0(0)};

    // send command to EV3 via Bluetooth
    int bt = open("/dev/tty.EV3-SerialPort", O_RDWR);
    write(bt, read_sensor, 13);

    // receive data back from EV3 via Bluetooth
    unsigned char sensor_data[255];
    read(bt, sensor_data, 255);
    for(int i=0; i<255; i++) {
        printf("%x", sensor_data[i]);
    }
    
    // end connection with EV3
    close(bt);
}

The structure of the code is pretty similar to what we had before. The first change is that now our command type is no longer DIRECT_COMMAND_NO_REPLY but DIRECT_COMMAND_REPLY, as we now want to receive data from the EV3. The second change is the sixth bytecode, which is now 1. That means we are now requesting one global variable - we’ll need it to store the sensor data.

The third change is of course the command itself, which is now opINPUT_READ. Its arguments are, in order: the EV3 brick (usually 0, unless you have multiple bricks), the port number (minus 1), the sensor type (0 = don’t change the type), and the sensor mode (0 = don’t change the mode). GV0 is not an argument, but the global variable where the sensor data will be stored. Like the motor power, the data we will get back will be in percentage (alternatively, there is an opINPUT_READSI command that returns the data in SI units).

The fourth change is that we now have a new code block. Its first line - unsigned char sensor_data[255] - creates an array of size 255, with all values initialized to zero. The size is 255 because at this point we don’t know exactly what the actual size of the received data will be, so we want to be safe: the data will be in hexadecimal format, so 255 is about as large as it gets (just as with the data we send, the first two bytes of the data we receive tell us how large the message is - but we can only count up to 255 with two bytes in hexadecimal format, so 255 is the limit here). The second line receives the data and the for loop prints each byte to the screen.

If everything went well you should see as output something like 400021F00000… Try it a couple more times, moving the sensor around in-between. You will notice that the first five digits or so don’t change, and neither do all the others after the sixth or seventh digit. For instance, your results will look like 400023D00000… or 400025B00000… Only two digits or so will change. That is your sensor data! In these three examples, for instance, your data points are 1F, 3D, and 5B. That’s hexadecimal format; in decimal format that means 31, 61, and 91 (here’s a conversion table). Now, once you’ve figured out what the relevant digits are you can get rid of that loop and print only them (say, printf("%x", sensor_data[5]);).

That’s it! Now you can control the motors and read the sensors - that should help you get started. If you inspect the c_com.h files you will see lots of other commands, some of them with usage examples. The way forward is by exploring the firmware code and by trial and error.

Happy building!

rise of the machines - part 1

Social scientists spend a lot of time writing code that has no physical manifestation. Other than changing pixels in a laptop screen our scripts do not change the material world in any observable way. Meanwhile our colleagues in the engineering department get to build Terminator-like robots and all sorts of fun machines. So I decided I wouldn’t live in perpetual envy anymore.

Enter LEGO Mindstorms.

I’ve been playing experimenting with it for about a week and it’s hard to overstate how much fun it is. Besides lots of regular LEGO pieces you get a mini-CPU, a couple of servomotors, a light sensor, and a proximity sensor. You plug the motors and sensors into the mini-CPU (usually referred to as the “EV3 brick”), use the regular LEGO pieces to assemble whatever robot your imagination produces, and then connect your computer to the EV3 brick to control the motors and read data from the sensors.

EV3 brick, motors, and sensors

The EV3 is the third generation of the LEGO Mindstorms set. The previous ones were called NXT and RCX. The EV3 improves on the NXT on several aspects, including processing power, memory size, number of ports, the inclusion of a microSD card slot, ability to communicate with iOS, and a larger screen. The NXT is still around though, but somehow it costs more than the EV3 ($449.99 vs $349.95 on Amazon); maybe that’s because there are more applications developed for the NXT, as it’s been around since 2006.

The EV3 brick runs on six AA batteries. They go flat really fast (two or three days with moderate use), so I strongly recommend that you buy rechargeable ones. Also, if you don’t want to have to disassemble the robot every time you need to recharge the batteries you should look into this.

Another thing you can do is use a Raspberry Pi instead of the EV3 brick. You can do it yourself if you’re familiar with these things or you can buy a Raspberry Pi that’s already customized for use with the EV3 sensors and motors (see here).

The first step is deciding how exactly you are going to interact with your robot. If you don’t want to write code, LEGO provides an app that lets you program visually, using flowcharts (a bit like in Scratch). You do the programming in your computer and then send the commands to the EV3 brick via USB, Bluetooth, or WiFi. It looks like this:

But more likely you will want to write code (if for no other reason than because the LEGO app is painfully slow - even in my quad-core laptop with 16GB of memory). You have a number of options.

Java

If Java is your cup of tea then leJOS is the way to go about it. leJOS is a Java Virtual Machine that can replace the native firmware of the EV3 brick and let you run Java code on it. It works like this: you get ahold of a microSD card, make it bootable (follow these instructions), insert it into the EV3 brick, and turn it on. If everything went well the EV3 will boot to leJOS (and not to its native firmware). You can then start writing your Java programs, using the leJOS API. (Conveniently, leJOS doesn’t mess with the native firmware: if you remove the microSD card and restart the EV3 it will boot to the native firmware again.)

leJOS initial screen

Python

The Python alternative works similarly to the Java one: you get ahold of a microSD card, make it bootable (following these instructions), insert it into the EV3 brick and turn it on. If everything went well the EV3 will boot the custom debian wheezy OS in the bootable card (and not to its native firmware). You then SSH into the EV3 and use this API to write your programs. (Same as with leJOS, the native firmware is still there for when you want it - just remove the microSD card and restart the EV3.)

The Python alternative sounds great, but somehow I couldn’t make it work. I insert the card, turn the EV3 on, but then nothing (visible) happens, it seems that at some point in the process the EV3 gets stuck. I tried re-flashing the card and also buying a different card, but nothing worked. After two days of failed attempts I gave up (I may come back to it at some point though). (Also note that even if it works the Bluetooth is not stable yet).

Matlab

There are two ways to program the EV3 using Matlab. One is the official EV3 package from MathWorks. Downside: it only works on Windows (32-bit and 64-bit) and on Linux (64-bit). Which takes us to the second alternative: the folks from CyPhy Lab have created a Matlab-EV3 toolkit that should work on Macs as well. Their website is full of documentation and examples to help you get started.

Unlike the Java and Python solutions the Matlab ones do not require any microSD cards. You will boot the EV3 normally and your program will communicate with the native EV3 firmware (via USB, Bluetooth, or WiFi). I think this is a huge plus.

Microsoft API

Microsoft has produced an API that lets you interact with the EV3 from Windows desktop, Windows Phone 8, and WinRT, using .NET, WinJS, and C++. (I don’t want to work with Windows, so I didn’t really look into it.)

C

The EV3’s native firmware is open source and written (largely) in C. If you clone it you can use C to communicate with the EV3. Like the Matlab and Microsoft solutions it doesn’t require any microSD cards; you just turn the EV3 on and communicate with its native firmware via USB, Bluetooth, or WiFi. And unlike the Matlab and Microsoft solutions it’s 100% open source - no need to spend money on Matlab or Windows. The downside is that the firmware source code is only partially documented, so figuring out the right command to do this or that involves a fair amount of trial and error (or asking for help on StackOverflow). But I think it’s still worth it (this is what I’ve been using).

This is it for now. In part 2 I’ll talk a bit about using C to communicate with the EV3. My end goal is to build a self-driving robot, for the fun of it. I’m not sure I’ll use C for everything; I may use it just to communicate with the EV3 and then pass all the data to Python for the machine learning algorithms. We’ll see.

This is something that I’m learning on-the-fly - I’ve never done any serious work in C before and I just learned what bytecode is -, so these posts will not be a tutorial, but merely a way to share these experiences (and perhaps receive some useful feedback from people who actually know this stuff).

language wars

Estonia

The Estonian online voting platform is open source and written in Python!

Here.

videos from PyCon 2014

Here (all of them).

Here are the stats-related ones:

Data intensive biology in the cloud: instrumenting ALL the things (Titus Brown)

Diving into Open Data with IPython Notebook & Pandas (Julia Evans)

Keynote (Fernando Pérez)

Know Thy Neighbor: Scikit and the K-Nearest Neighbor Algorithm (Portia Burton)

Bayesian statistics made simple (Allen Downey)

Beyond Defaults: Creating Polished Visualizations Using Matplotlib (Hannah Aizenman)

Data Wrangling for Kaggle Data Science Competitions – An etude (Krishna Sankar)

Enough Machine Learning to Make Hacker News Readable Again (Ned Jackson Lovely)

Hands-on with Pydata: how to build a minimal (Christian Fricke , Diego Maniloff , Zach Howard)

How to Get Started with Machine Learning (Melanie Warrick)

IPython in depth: high productivity interactive and parallel python (Fernando Perez)

Python for Social Scientists (Renee Chu)

Python + Geographic Data = BFFs (Mele Sax-Barnett)

Realtime predictive analytics using scikit-learn & RabbitMQ (Michael Becker)

Diving deeper into Machine Learning with Scikit-learn (Jake Vanderplas , Olivier Grisel)

Exploring Machine Learning with Scikit-learn (Jake Vanderplas , Olivier Grisel)

How to formulate a (science) problem and analyze it using Python code (Eric Ma)

And here are the webscraping/text_processing-related ones:

Character encoding and Unicode in Python (Esther Nam , Travis Fischer)

Introduction to Regular Expressions (Luke Sneeringer)

Introduction to Web (and data!) Scraping with Python (Katharine Jarmul)

Python Scraping Showdown: A performance and accuracy review (Katharine Jarmul)

Search 101: An Introduction to Information Retrieval (Christine Spang)

Mining Social Web APIs with IPython Notebook (Matthew Russell)