5 reasons why academics should read Anathem

I just read Neal Stephenson’s 2008 novel Anathem and now I walk around pestering everyone I know telling them to read it too. Well, not everyone: just people who are or have been in academia. Judging from Goodreads reviews everyone else finds the novel too long and theoretical, full of made up words, and full of characters who are too detached from the real world to be believable. Pay no heed to the haters - here’s why you academic types should read Anathem:

1. You will feel right at home even though the story is set in an alien world (I).

The planet is called Arbre and its history and society are not radically different from Earth’s. Except that at some point (thousands of years before the story begins) the people of Arbre revolted against science and confined their intellectuals to monasteries where the development and use of technology is severely limited - no computers, no cell phones, no internet, no cameras, etc -, as is any contact with the outside world. Inside these monasteries (“Maths”) the intellectuals (the “avout”) dedicate themselves to the study and development of mathematics, physics, and philosophy. As the use of technology is restricted, all that research is purely theoretical.

Arbre’s Maths are therefore an allegory for Earth’s universities. How many of our papers and dissertations end up having any (non-academic) impact? Maybe 1% of them? Fewer than that? In (Earth’s) academia the metric of success is usually peer-reviewed publications, not real-world usefulness. Even what we call, say, “applied econometrics” or “applied statistics” is more often than not “applied” only in a limited, strictly academic sense; when you apply econometrics to investigate the effect of economic growth on democracy that is unlikely to have any detectable effect on economic growth or democracy.

So, in Anathem you find this bizarre alien world where intellectuals are physically confined and isolated from the rest of the world and can’t use technology and yet that world feels familiar and as a (current or former) scholar you won’t react to that in the same way other people do. If you go check the reviews on Goodreads you’ll see lots of people complaining that the Maths are unrealistic. To you, however, Maths will sound eerily natural; Anathem would be more alien to you if the Maths were, say, engineering schools.

(Needless to say, the allegory only goes so far, as Arbran’s avout are legally forbidden from having any real-world impact; having no choice in the matter, they don’t lose any sleep over the purely academic nature of their work. And of course people do produce lots of useful research at Earth’s universities.)

2. You will feel right at home even though the story is set in an alien world (II).

The way an Arbran avout progresses in his or her mathic career is entirely different from the way an Earthly scholar progresses in his or her academic career - and yet way too familiar. In Arbre you start by being collected at around age 10. That makes you a “fid” and you will be mentored and taught by the more senior avout, each of which you will respectully address as “pa” or “ma”. When you reach your early twenties you choose - and are chosen by - a specific mathic order. There are many such orders, each named after the avout who founded it - there are the Edharians, the Lorites, the Matharrites, and so on, each with specific liturgies and beliefs.

The avout are not allowed to have any contact with the outside world (the “extramuros”) except at certain regular intervals: one year (the Unarian maths), ten years (the Decenarian maths), one hundred years (the Centenarian maths), or one thousand years (the Milleniarian maths). And only for ten days (those days are called “Apert”). You can get collected by any math - Unarian, Decenarian, Centenarian, or Millenarian. If you get collected, say, at a Unarian math, and you show a lot of skill and promise, you can get upgraded (“Graduated”) to a Decenarian math. If you keep showing skill and promise you can get Graduated to a Centenarian math. And so on. The filter gets progressively stricter; only very few ever get Graduated to the Millenarian maths.

So, the reward for being isolated from the outside world and focusing intensely on your research is… getting even more isolated from the outside world so that you can focus even more intensely on your research. Sounds familiar?

3. Anathem gives you vocabulary for all things academia.

Think back to your Ph.D. years and remember the times you went out with your fellow fids for drinks (well, if you were actual fids you wouldn’t be able to leave your math - you could, but then you wouldn’t be able to go back, except during Apert - but never mind that). Weird conversations (from the point of view of those overhearing them) ensued and you got curious looks from waiters and from other customers.

Why? Because you spoke in the jargon of your field - you used non-ordinary words and you used ordinary words in non-ordinary ways. Like “instrumental” or “endogeneity” or “functional programming”. Not only that: the conversations were speculative and obeyed certain unwritten rules, like Occam’s razor. Clearly these were not the same conversations you have with non-avout - your college friends, your family, your Tinder dates. And yet you call all of them “conversations”. Well, not anymore; Anathem gives you a word for inter-avout conversation about mathic subjects: Dialog. Neal Stephenson goes as far as creating a taxonomy of Dialog types:

Dialog, Peregrin: A Dialog in which two participants of roughly equal knowledge and intelligence develop an idea by talking to each other, typically while out walking around.

Dialog, Periklynian: A competitive Dialog in which each participant seeks to destroy the other’s position (see Plane).

Dialog, Suvinian: A Dialog in which a mentor instructs a fid, usually by asking the fid questions, as opposed to speaking discursively.

Dialog: A discourse, usually in formal style, between theors. “To be in Dialog” is to participate in such a discussion extemporaneously. The term may also apply to a written record of a historical Dialog; such documents are the cornerstone of the mathic literary tradition and are studied, re-enacted, and memorized by fids. In the classic format, a Dialog involves two principals and some number of onlookers who participate sporadically. Another common format is the Triangular, featuring a savant, an ordinary person who seeks knowledge, and an imbecile. There are countless other classifications, including the suvinian, the Periklynian, and the peregrin.

(Anathem, pp. 960-961)

(Yes, there is a glossary in Anathem.)

You can’t get much more precise than that without being summoned to a Millenarian math.

Dialog is just one example. You left academia? You went Feral.

Feral: A literate and theorically minded person who dwells in the Sæculum, cut off from contact with the mathic world. Typically an ex-avout who has renounced his or her vows or been Thrown Back, though the term is also technically applicable to autodidacts who have never been avout.

(Anathem, p. 963)

You left academia to go work for the government? You got Evoked.

Voco: A rarely celebrated aut by which the Sæcular Power Evokes (calls forth from the math) an avout whose talents are needed in the Sæcular world. Except in very unusual cases, the one Evoked never returns to the mathic world.

(Anathem, p. 976)

Reviewer #2 says your argument is not original? He’s a Lorite.

Lorite: A member of an Order founded by Saunt Lora, who believed that all of the ideas that the human mind was capable of coming up with had already been come up with. Lorites are, therefore, historians of thought who assist other avout in their work by making them aware of others who have thought similar things in the past, and thereby preventing them from re-inventing the wheel.

(Anathem, p. 967)

Got friends or family who are not academics? Well, ok, J. K. Rowling has already given us a word for that - muggles. But in some languages that word gets super offensive translations - in Brazilian Portuguese, for instance, they made it “trouxas”, which means “idiots”. Not cool, Harry Potter translators. But worry not, Neal Stephenson gives us an alternative that’s only a tiny bit offensive: “extras” (from “extramuros” - everything outside the maths).

Extra: Slightly disparaging term used by avout to refer to Sæcular people.

(Anathem, p. 963)

That cousin of yours who believes the Earth is flat? He is a sline.

Sline: An extramuros person with no special education, skills, aspirations, or hope of acquiring same, generally construed as belonging to the lowest social class.

(Anathem, p. 973)

And of course, what happens to a scholar who gets expelled from academia? He gets anathametized.

Anathem: (1) In Proto-Orth, a poetic or musical invocation of Our Mother Hylaea, used in the aut of Provener, or (2) an aut by which an incorrigible fraa or suur is ejected from the mathic world.

(Anathem, pp. 956-957)

And so on and so forth. Frankly, it’s amazing that academics manage to have any Dialogs whatsoever without having read Anathem.

(I must note that Neal Stephenson not only puts these words in the book’s glossary, he uses them extensively throughout the book - there are 40 occurrences of “evoked”, 90 occurrences of “Dialog”, and 57 occurrences of “sline”, for instance. And because there is a glossary at the end he doesn’t bother to define these words in the main text, he just uses them. Which can make your life difficult if, like me, you didn’t bother to skim the book before reading it and only found out about the glossary after you had finished. Damn Kindle.)

4. Anathem might be the push you need to quit social media for good.

I’ve been reading Cal Newport’s Deep Work, about the importance of focusing hard and getting “in the zone” in order to be productive. (Well, “reading” is inaccurate. I bought the audio version and I’ve been listening to it while driving - which is not without irony.) There isn’t a whole lot of novelty there - it’s mostly common sense advice about “unplugging” for at least a couple of hours each day so you can get meaningful work done (meaningful work being work that imposes some mental strain, as opposed to replying emails or attending meetings). The thing is, at a certain point, much to my amusement and surprise, Cal Newport mentions Neal Stephenson.

As Cal Newport tells us, Neal Stephenson is a known recluse. He doesn’t answer emails and he is absent from social media. To Newport, that helps explain Stephenson’s productivity and success (No, I won’t engage you in a long Periklynian Dialog about how we can’t establish causality based on anecdotal evidence. That’s not the point and in any case Cal Newport, despite being an avout himself - he’s a computer science professor at Georgetown - is trying to reach an audience of extras and Ferals.) I had read other Neal Stephenson books before - Cryptonomicon, Snow Crash, The Diamond Age, REAMDE, Seveneves -, but I had never bothered to google the man, so I had no idea how he lived. After Cal Newport’s mention, though, I think Anathem is a lot more personal than it looks. Among its many messages maybe there is Neal Stephenson telling us “see? this is what can be achieved when smart people are locked up and cut off from the world”. “What can be achieved” being, in Neal Stephenson’s case (and brilliantly recursively), a great novel about what can be achieved when smart people are locked up and cut off from access to the world.

5. Anathem may be an extreme version of what happens when people turn against science.

Flat-Earthers and anti-vaxxers are back. People who don’t know what a standard-deviation is pontificate freely and publicly about the scientific evidence of climate change. Violent gangs openly oppose free speech at universities. I’m not saying these slines are about to lock up Earth’s scientists in monasteries, but perhaps the Temnestrian Iconography is getting more popular.

“[…] Fid Erasmas, what are the Iconographies and why do we concern ourselves with them?” […]

“Well, the extras—”

“The Sæculars,” Tamura corrected me.

“The Sæculars know that we exist. They don’t know quite what to make of us. The truth is too complicated for them to keep in their heads. Instead of the truth, they have simplified representations— caricatures— of us. Those come and go, and have done since the days of Thelenes. But if you stand back and look at them, you see certain patterns that recur again and again, like, like— attractors in a chaotic system.”

“Spare me the poetry,” said Grandsuur Tamura with a roll of the eyes. There was a lot of tittering, and I had to force myself not to glance in Tulia’s direction. I went on, “Well, long ago those patterns were identified and written down in a systematic way by avout who make a study of extramuros. They are called Iconographies. They are important because if we know which iconography a given extra— pardon me, a given Sæcular— is carrying around in his head, we’ll have a good idea what they think of us and how they might react to us.”

Grandsuur Tamura gave no sign of whether she liked my answer or not. But she turned her eyes away from me, which was the most I could hope for. “Fid Ostabon,” she said, staring now at a twenty-one-year-old fraa with a ragged beard. “What is the Temnestrian Iconography?”

“It is the oldest,” he began. “I didn’t ask how old it was.” “It’s from an ancient comedy,” he tried.

“I didn’t ask where it was from.”

“The Temnestrian Iconography…” he rebegan.

“I know what it’s called. What is it?

“It depicts us as clowns,” Fraa Ostabon said, a little brusquely. “But… clowns with a sinister aspect. It is a two-phase iconography: at the beginning, we are shown, say, prancing around with butterfly nets or looking at shapes in the clouds…”

“Talking to spiders,” someone put in. Then, when no reprimand came from Grandsuur Tamura, someone else said: “Reading books upside-down.” Another: “Putting our urine up in test tubes.”

“So at first it seems only comical,” said Fraa Ostabon, regaining the floor. “But then in the second phase, a dark side is shown— an impressionable youngster is seduced, a responsible mother lured into insanity, a political leader led into decisions that are pure folly.”

“It’s a way of blaming the degeneracy of society on us— making us the original degenerates,” said Grandsuur Tamura. “Its origins? Fid Dulien?”

“The Cloud-weaver, a satirical play by the Ethran playwright Temnestra that mocks Thelenes by name and that was used as evidence in his trial.”

“How to know if someone you meet is a subscriber to this iconography? Fid Olph?”

“Probably they will be civil as long as the conversation is limited to what they understand, but they’ll become strangely hostile if we begin speaking of abstractions…?”

(Anathem, pp. 71-72)


This is it. Go read Anathem and tell your fellow avout and Ferals about it. See you at Apert.

doing data science when you live in a failed state

Brazil is the undisputed world leader in homicides: over 50 thousand a year, which is more than Europe, Oceania, United States, Russia, and China combined. Yes, combined. Yes, the whole freaking Europe. Yes, the supposedly gun-loving United States. Yes, China with its 1.3 billion people. Brazil beats these continents and countries by 4,473 homicides, which is roughly equivalent to Uganda or to ten Canadas. No, I’m not making these numbers up. Take a moment to let that sink in.

As you might guess, a country with lots of homicides also tends to have lots of robbery. I’d love to take my MacBook Pro to a coffee shop and work there all day like I used to when I was in grad school - back when I lived in lovely, safe, Columbus, Ohio. But if I do that in Brasília I’ll probably come back home empty handed (if I come back home at all). You can’t parade Apple gear around when you live in a failed state.

I finally got tired of working from home all weekend, so I decided to enable SSH and HTTP connections into my home network, so I can use my Mac remotely as if it were an AWS server. That way I can go to the coffee shop with my old, cheap Lenovo - or even a tablet or smartphone - and use it to connect to my MacBook, which will remain safe and sound back home. It took some doing and I imagine others may be going through the same problem (i.e., wanting to work at a coffee shop but living in an episode of The Walking Dead), so here’s a how-to.

My setup is: Humax HG100R-L2 modem (that’s what most clients of NET - Brazil’s largest cable company - have), AirPort Extreme Base Station router, MacBook Pro. Your setup will likely differ, but you can probably tweak the instructions here to fit whatever you have.

step 1: your modem

If you have both a modem and a router then the easiest way to go about this is to put your modem in ‘bridge mode’. That means disabling your modem’s advanced settings and delegating them to your router. That way you only need to worry about router settings. You won’t need to worry about complex interactions between your modem settings and router settings.

Head to http://192.168.0.1/ on your browser. You should see the page below.

If you’ve never changed them, your id and password are ‘admin’ and ‘password’ respectively. Sign in. You should see the following, except with your WiFi network name and password shown under “SSID(2.4GHz)” and “Senha” respectively. (Your password will be shown in plain characters, not as a bunch of dots, so don’t let your neighbors peek.) (Yes, Humax’ settings are in a mix of Portuguese and English. It beats me too.)

Click “Advanced Network Settings” (lower right corner). You should see something like this:

Click on “Definir” (between “Status” and “Back Up”, second column from the left). You should see a page with a bit more options than the following one (that’s because your modem is not in bridge mode yet).

On the “Modo Switch” menu, choose “Bridge”, then click “Aplicar”. Click “ok” on whatever confirmation pop up appears. This will make you go offline for a couple of minutes, as your modem resets itself. Wait until it’s back up online again and voilà, your modem is now in bridge mode.

(If you ever need to tweak your modem settings again, it’s no longer http://192.168.0.1/ but http://192.168.100.1)

step 2: your router

On to your router now. We need to tell it to accept incoming SSH and HTTP connections. In order to do that we need to tell your router to map those types of connections to specific ports.

On your Mac, open the AirPort Utility app.

Click on the AirPort Extreme picture to go into your routers’ settings and go to the ‘Network’ tab. You should see something like this:

We’ll make a lot of changes here. First, on the “Router Mode” dropdown menu, choose “DHCP and NAT” if that’s not the chosen value already. Then click the “+” button near “DHCP Reservations”. That will open a small page. You’ll make it look like the one below by selecting the exact same choices. (To do that you’ll need to know your MAC address, which you can find out in your Mac by going into “System Preferences”, “Network”, “Advanced”; it’s the combination of digits you see right next to “Wi-Fi Address”.) When everything matches, click “Save”.

Now you’re back to this:

Click the “+” button near “Port Settings”. A small page will pop up. Tweak all the fields so that it looks exactly like this:

Click “Save”. Then click the “+” button near “Port Settings” again. The same small page will pop up. Make it look exactly like this:

Click “Save”. Then click “Update”. Your router will go crazy for a moment as it does its magic. Wait until it comes back up online and voilà, you have allowed SSH and HTTP connections into your home network. SSH connections will be forwarded to port 22 and HTTP connections will be forwarded to port 8080.

step 3: your Mac

This part is simple. Go to “System Preferences”, “Sharing”, and enable Remote Login:

If your firewall is active then you need to tell it to allow incoming traffic through ports 22 and 8080. This can be a bit tricky and it depends on your OS version. This may help. Alternatively, you can take the lazy and insecure path of simply disabling your firewall altogether (“System Preferences”, “Security and Privacy”, “Firewall”).

step 4: your IP address

You need to know your MacBook’s public IP address so you can access it from the outside. This should tell you. Write it down.

My experience with NET in Brazil (and with TimeWarnerCable in the US) is that IP addresses don’t change that often. But they do sometimes. If that bothers you you may ask that your cable provider give you a static IP address (they may charge a small fee for that). (EDIT: alternatively, you can use a Dynamic DNS service - like this; h/t Thompson Marzagão.)

step 5: your coffee shop

Take whatever cheap, inconspicous piece of hardware you have at hand to your favorite coffee shop. Launch a terminal and do ssh myusername@myipaddress, where myusername is the username you normally use to log into your Mac and myipaddress is the IP address you wrote down in step 4. Enter your password and that’s it, you are now inside your Mac. You can cd into different directories, run code, do whatever you want.

If your coffee shop hardware is a tablet or smartphone, Termius is a terrific SSH client for mobile devices.

step 6 (optional): your data science

Wondering why I made you enable HTTP connections? Well, here comes the really fun part: Jupyter notebooks. You can start a Jupyter server in your Mac and then, with your coffee shop cheapoware, use your browser to write code interactively and have it run on your Mac. Jupyter’s default language is Python but you can install kernels for an increasingly large number of languages, like R and Julia.

On your Mac, do pip install jupyter to install Jupyter and then do jupyter notebook --ip='0.0.0.0' --port='8080' --no-browser to start the Jupyter server. You’ll be given a url. Something like http://0.0.0.0:8080/?token=sfdsfs90809809s8dfs0df8sdf. Replace 0.0.0.0 by myipaddress (see step 4). That’s the address you’ll use at the coffee shop to launch Jupyter notebooks.

(If your cheapoware is a laptop things should work right out-of-the-box. If it’s an iOS device then you have some additional steps to take - see here.)

step 7: your venti caramel macchiato

That’s it! You have now reduced your likelihood of getting mugged and minimized your losses in case you do get mugged. Time to grab your katana and go mingle with the locals.

using deep learning to detect fake exports

New paper. Abstract:

Normally exports of goods and products are transactions encouraged by the governments of countries. Typically these incentives are promoted by tax exemptions or lower tax collections. However, exports fraud may occur with objectives not related to tax evasion, for example money laundering. This article presents the results obtained in implementing the unsupervised Deep Learning model to classify Brazilian exporters regarding the possibility of committing fraud in exports. Assuming that the vast majority of exporters have explanatory features of their export volume which interrelate in a standard way, we used the AutoEncoder to detect anomalous situations with regards to the data pattern. The databases used in this work come from exports of goods and products that occurred in Brazil in 2014, provided by the Secretariat of Federal Revenue of Brazil. From attributes that characterize export companies, the model was able to detect anomalies in at least twenty exporters.

Django for Flask users

I’m using Django for a serious project for the first time. I had played with Django a couple of times before, but I’m a long-time Flask fanboy and I invariably gave up in frustration (“why would anyone ever need separate files for settings, urls, and views?!”). Well, turns out Django is pretty cool if you want to put a bunch of apps under the same umbrella. Now, the official tutorial is a bit too verbose if you’re impatient. And if you’re used to Flask’s minimalism, you will get impatient with Django at times. So, here a few potentially useful pointers (largely for my own future consultation).

To get started, just pip install Django, run django-admin startproject mysite, then run python manage.py startapp myapp. (Replace mysite and myapp by whatever names you want.) This should create the essential files and directories you’ll need.

making urls work

In Flask you create your views and map your urls all at once:

@app.route('/')
def index():
    return 'Hello World!'

This is about as simple as it gets (unless you want to get really minimalist).

In Django you can’t do that. You have to define your views in one place and map your urls elsewhere. The usual way to do it is to define your views in your (aptly named) myapp/views.py file, like this:

from django.http import HttpResponse

def index(request):
    return HttpResponse('Hello World!')

Unlike in Flask you can’t just do return 'Hello World!' - the returned object cannot be a string, so we need to import HttpResponse. Also unlike in Flask, we must feed the request to the function - there is no global request object in Django, so we need to pass it around explicitly (more about this in a moment).

Now on to mapping urls. This requires changing two different files. The first is your mysite/urls.py file, wherein you’ll put this:

from django.conf.urls import url

urlpatterns = [url(r'^myapp/', include('myapp.urls'))]

This piece of code tells mysite (the big project inside which your various apps will live) to defer to myapp (one of your various apps) whenever someone hits http://blablabla/myapp/. (That r'^myapp/ thing is a regular expression that matches any url that contains myapp/.)

So, mysite/urls.py is a big dispatcher: it’ll check the url and send the request to the appropriate app. Here we only have one app (myapp), but if you’re using Django you’ll likely have several apps, in which case the urlpatterns list will contain several url() objects.

Now, myapp must be prepared to receive the baton. For that to happen your myapp/urls.py file (not your mysite/urls.py file!) must look like this:

from django.conf.urls import url
from . import views

urlpatterns = [url(r'^$', views.index, name = 'index')]

Here we have another regex: r'^$. This will capture any requests that end in myapp/. (If the request got this far then it must contain myapp/, so you don’t need to repeat it in the regex here.) We’re telling myapp that any such requests should be handled by the view function named index - which you defined before, in your myapp/views.py file (see above).

So, myapp/urls.py is a secondary dispatcher: it’ll check the url and send the request to the appropriate view. Here we only have one view (the app’s index page), but in real life you’ll have several views, in which case the urlpatterns list will contain several url() objects.

That’s it. If you run python manage.py runserver and then open http://127.0.0.1:8000/ in your browser you should be greeted by the Hello World! message.

If you really want to you can have a single-file Django project: check this. But if your project is so small that you can have a single file then maybe you’d be better off using Flask or CherryPy or some other minimalist web framework.

request and session

Accessing request and session data in Flask is a no brainer. There is a global request object and a global session object and, well, you just do whatever you want to do with them.

from flask import request
from flask import session

@app.route('/')
def hello():
    if request.method == 'POST':
        user_input = request.form['user_input']
        session['foo'] = 'bar'
    elif request.method == 'GET':
        session['foo'] = 'macarena'
    return session['foo']

In Django, as I mentioned before, there is no global request object - you need to explicitly pass request around to work with it. There is no global session object either. Instead, session is an attribute of request. This is how the above snippet translates into Django:

from django.http import HttpResponse

def hello(request):
    if request.method == 'POST':
        user_input = request.POST['user_input']
        request.session['foo'] = 'bar'
    elif request.method == 'GET':
        request.session['foo'] = 'macarena'
    return HttpResponse(request.session['foo'])

So, session becomes request.session and request.form becomes request.POST.

templates

You must tell Django where to look for templates. Open mysite/settings.py, locate the TEMPLATES list and edit DIRS.

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': ['/path/to/my/templates/folder/',
                 '/path/to/my/other/templates/folder/'],
        'APP_DIRS': True,
        'OPTIONS': {
            # ...
        },
    },
]

There are a few syntax differences between Jinja2 (Flask’s default templating language) and Django’s templating language (DTL). For instance, to access the first element of mylist it’s {% mylist[0] %} in Jinja2 but {% mylist.0 %} in DTL. But most of the syntax is identical. Template inheritance works the same way, with {% extends 'parent.html' %} and {% block blockname %}{% endblock $}. Same with loops, if/elses, and variables: {% for bla in blablabla %}{% endfor %}, {% if something %}{% elif somethingelse %}{% else %}{% endif %}, {{ some_variable }}. If you’re porting something from Flask to Django there is a chance your templates will work just as they are.

You need to change your views though. In Flask you render a template and pass variables to it like this:

from flask import render_template

@app.route('/')
def hello():
    return render_template('mytemplate.html', 
                           some_var = 'foo', 
                           other_var = 'bar')

In Django you do it like this:

from django.shortcuts import render

def hello(request):
    return render(request,
                  'mytemplate.html', 
                  {'some_var' = 'foo', 
                   'other_var' = 'bar'})

So, in Django you must pass the request object to render the template. And your template variables must be passed as a dict.

connections

In both Flask and Django you can use something like pyodbc or pymssql to connect to your databases. But you can put a layer of abstraction on top of that. In Flask there is Flask-SQLAlchemy. Here’s their quickstart snippet:

from flask import Flask
from flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True)
    email = db.Column(db.String(120), unique=True)

    def __init__(self, username, email):
        self.username = username
        self.email = email

    def __repr__(self):
        return '<User %r>' % self.username

In Django the connection and the models go into separate scripts. You set up the connection by adding an entry to the DATABASES dict in your mysite/settings.py file:

DATABASES = {

    # ... your other db connections ...

    'my_database_name': {
        'ENGINE': 'django.db.backends.sqlite3', 
        'NAME': 'my_database_name',
        'USER': 'my_username',
        'PASSWORD': 'my_password',
        'HOST': 'my.host.address',
        'PORT': 'my_port'}
    }

Then, in your myapp/models.py, you define your models.

from django.db import models

class User(models.Model):
    id = models.IntegerField()
    username = models.CharField(max_length = 80)
    email = models.CharField(max_length = 120)

You don’t have to use any models though. If you prefer to run raw SQL queries you can do it like this:

from django.db import connections

cursor = connections['my_database_name'].cursor()
cursor.execute('SELECT * FROM sometable')
results = cursor.fetchall()

Just as you would do with pyodbc (except that here you don’t need to .commit() after every database modification).

afterwards

I’m just trying to get you past the initial rage over all the boilerplate code Django requires. :-) This is all just about syntax - I’m merely “translating” Flask to Django. If you’re serious about Django you should invest some time in learning Django’s semantics. Their official tutorial is a good place to start. Have fun!

I need to use Google App Engine to text my girlfriend

This is the story of how I had to build and deploy a freaking app just so I can text my girlfriend when I’m at the office. Perhaps it’ll help others who are also subject to the arbitrary rules of IT departments everywhere. (Dilberts of the world, unite!)

For some two years now my messaging app of choice has been Telegram. It’s lightweight, end-to-end encrypted, well designed, and free; it’s impossible not to love it. Now, I hate typing on those tiny on-screen keyboards, so most of the time what I actually use is Telegram’s desktop app. Problem is, I can’t use it when I’m at work. My organization’s IT department blocks access to Telegram’s servers (dont’ ask). I can install the app, but it doesn’t connect to anything; it can’t send or receive messages.

So, I looked into Telegram’s competitors. I tried WhatsApp, but its desktop version is blocked as well at my organization. And in any case I tried it at home and it’s sheer garbage: the desktop app needs your phone to work (!) and it crashes every ~15 minutes. (I keep pestering my friends to switch from WhatsApp to Telegram but WhatsApp is hugely popular in Brazil and network externalities get in the way.)

Then it hit me: why not Slack? The IT department doesn’t block it and I already use Slack for professional purposes. Why not use it to talk to my girlfriend too? I created a channel, got her to sign up, and we tried it for a couple of days.

Turns out Slack solved the desktop problem at the cost of creating a mobile problem. I don’t have any issues with Slack’s web interface - I keep my channels open on Chrome at all times and that works just fine. But when I switch to mobile… boy, that’s one crappy iOS app. Half the time it just doesn’t launch. Half the time it takes forever to sync. Granted, my iPhone 5 is a bit old. But the Telegram iOS app runs as smooth and fast as it did two years ago, so the hardware is not at fault here.

As an aside, turns out Slack’s desktop app is also ridiculously heavy. I don’t really use it - I use Slack’s web interface instead -, but that’s dispiriting nonetheless.

I tried Facebook’s Messenger. Blocked. I tried a bunch of lesser-known alternatives. Blocked.

Eventually I gave up on trying different messaging apps and asked the IT department to unblock access to Telegram’s servers. They said no - because, well, reasons. (In the words of Thomas Sowell, “You will never understand bureaucracies until you understand that for bureaucrats procedure is everything and outcomes are nothing”.)

The IT guys told me I could appeal to a higher instance - some committee or another -, but I’ve been working in the government for a while and I’ve learned to pick my fights. Also, I believe in Balaji Srinivasan’s “don’t argue” policy.

So, I rolled up my sleeves and decided to build my own solution.

I don’t need to build a full-fledged messaging app. What I need is extremely simple: a middleman. Something that serves as a bridge between my office computer and Telegram’s servers. I need a web app that my office computer can visit and to which I can POST strings and have those strings sent to my girlfriend’s Telegram account.

That app needs to be hosted somewhere, so the first step is choosing a platform. I briefly considered using my personal laptop for that, just so I didn’t have to deal with commercial cloud providers. But I worry about exposing to the world my personal files, laptop camera, browser history, and the like. Also, I want 24/7 availability and sometimes I have to bring my laptop to the office.

I settled on Google App Engine. I used it before (to host an app that lets people replicate my Ph.D. research) and I liked the experience. And, more importantly, it has a free tier. GAE has changed quite a bit since the last time I used it (early 2014), but it has an interactive tutorial that got me up to speed in a matter of minutes.

You can choose a number of programming languages on GAE. I picked Python because that’s what I’m fastest at. (In hindsight, perhaps I should’ve used this as a chance to learn some basic Go.)

Instead of starting from scratch I started with GAE’s default “Hello, world!” Python app. The underlying web framework is Flask. That’s my go-to framework for almost all things web and that made things easier. Using Flask, this is how you control what happens when a user visits your app’s homepage:

# this is all in the main.py file of GAE's default "Hello, world!" Python app
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello, world!'

I don’t want a static webpage though, I want to communicate with Telegram’s servers. In order to do that I use a Python module called telepot. This is how it works: you create a Telegram bot account and then you use telepot to control that bot. (In other words, the sender of the messages will not be you, it will be the bot.

When you create your bot you receive a token credential, which you will then pass to telepot.

import telepot
bot = telepot.Bot('YOUR_TOKEN')
bot.getMe()

You can now make your bot do stuff, like sending messages. Now, Telegram enforces a sort of Asimovian law: a bot cannot text a human unless it has been texted by that human first. In other words, bots can’t initiate conversations. So I created my bot, told my girlfriend its handle (@bot_username), and had her text it. That message (like all Telegram messages) came with metadata (see here), which included my girlfriend’s Telegram ID. That’s all I need to enable my bot to text her.

girlfriend_id = 'SOME_SEQUENCE_OF_DIGITS'
bot.sendMessage(girlfriend_id, 'How you doing?')

Now let’s merge our web app code and our telepot code in our main.py file:

import telepot
from flask import Flask
app = Flask(__name__)

bot = telepot.Bot('YOUR_TOKEN')
bot.getMe()
girlfriend_id = 'SOME_SEQUENCE_OF_DIGITS'

@app.route('/')
def textGirlfriend():
    bot.sendMessage(girlfriend_id, 'How you doing?')
    return 'message sent!'

(This can be misused in a number of ways. You could, say, set up a cron job to text ‘thinking of you right now!’ to your significant other at certain intervals, several times a day. Please don’t.)

The rest of the default “Hello, world!” Python app remains the same except for two changes: a) you need to install telepot; use pip install with the -t option to specify the lib directory in your repository; and b) you need to add ssl under the libraries header of your app.yaml file.

So, I created a web app that my IT department does not block and that texts my girlfriend when visited. But I don’t want to text ‘How you doing?’ every time. So far, the app doesn’t let me choose the content of the message.

Fixing that in Flask is quite simple. We just have to: a) add a text field to the homepage; b) add a ‘submit’ button to the homepage; and c) tell the app what to do when the user clicks ‘submit’. (We could get fancy here and create HTML templates but let’s keep things simple for now.)

import telepot
from flask import Flask
from flask import request # so that we can get the user's input
app = Flask(__name__)

bot = telepot.Bot('YOUR_TOKEN')
bot.getMe()
girlfriend_id = 'SOME_SEQUENCE_OF_DIGITS'

@app.route('/')
def getUserInput():
    return '<form method="POST" action="/send"><input type="text" name="msg" size="150"><br><input type="submit" value="submit"></form>'

@app.route('/send', methods = ['POST'])
def textGirlfriend():
    bot.sendMessage(girlfriend_id, request.form['msg'])
    return 'message sent!'

And voilà, I can now web-text my girlfriend.

Yeah, I know, that would hardly win a design contest. But it works.

This is where I’m at right now. I did this last night, so there is still a lot of work ahead. Right now I can send messages this way, but if my girlfriend simply hit ‘reply’ her message goes to the bot’s account and I just don’t see it. I could have the app poll the bot’s account every few seconds and alert me when a new message comes in, but instead I think I’ll just create a Telegram group that has my girlfriend, myself, and my bot; I don’t mind reading messages on my phone, I just don’t like to type on my phone. Another issue is that I want to be able to text-app my family’s Telegram group, which means adding radio buttons or a drop-down menu to the homepage so I can choose between multiple receivers. Finally, I want to be able to attach images to my messages - right now I can only send text. But the core is built; I’m free from the tyranny of on-screen keyboards.

This is it. In your face, IT department.