In part 2 we saw how to use C++ and bytecodes to program LEGO Mindstorms EV3 bricks. Now, bytecodes don’t make for human-readable scripts. And C++ scripts take ~4 times longer to write than equivalent Python or Perl scripts. So, I’ve started writing a Python module that should make life easier - I’ve called it ev3py.
Here’s the GitHub repo. For now the module is still inchoate; it only covers three basic functions (starting motors, stopping motors, and reading data from sensors) and it only works on Macs, and only via Bluetooth. But it’s a start.
Let’s see a concrete example. Say you want to start the motor on port A with power 20. If you’re using bytecodes and C++ you need to write something like this:
So, with ev3py the code becomes human-readable and intuitive. It also becomes much faster to write. You no longer need to set message size, message counter, command type, etc.
Unlike other EV3 modules ev3py interacts with the EV3’s native firmware, so there’s no need to make the EV3 boot to a different operating system; just turn the brick on and you’re ready.
The goal is to eventually cover all EV3 capability and make ev3py work with USB and WiFi and also with Linux and Windows. I.e., something along the lines of the QUT-EV3 toolkit and the Microsoft EV3 API. If you’d like to contribute your help is much appreciated - just fork the GitHub repo and add capabilities, fix bugs, or suggest changes to the overall structure of the module.
Here I show how to use C++ to communicate via Bluetooth with the LEGO Mindstorms EV3 brick (see previous post).
If you are on a Mac everything should work right away. If you are using Ubuntu or other Linux distro I think you’ll only need to change the Bluetooth part a bit (my Ubuntu laptop doesn’t have Bluetooth, so I can’t be sure). If somehow you are forced to use Windows I think you’ll need to change the Bluetooth part a lot. All the rest should be the same though.
So, you start by cloning the source code of the EV3 firmware: open up your terminal and do git clone https://github.com/mindboards/ev3sources.git Name the folder ev3sources, to make the examples below easier to run. Also, open the ev3sources/lms2012/c_com/source/c_com.h file and change the line #include "lms2012.h" in order to provide the full path to the lms2012.h file. Say:
That’s all the setup you need - you are now ready to write and send commands to the EV3. Turn on your EV3, enable Bluetooth, make it discoverable (see the EV3 user guide if necessary), plug some motor to port A, fire up Xcode or whatever IDE you use, and try running the following code snippet:
#include<unistd.h>
#include<fcntl.h>
#include"ev3sources/lms2012/c_com/source/c_com.h"intmain(){// write command to start motor on port A with power 20unsignedconstcharstart_motor[]{13,0,0,0,DIRECT_COMMAND_NO_REPLY,0,0,opOUTPUT_POWER,LC0(0),LC0(1),LC1(20),opOUTPUT_START,LC0(0),LC0(1)};// send command to EV3 via Bluetoothintbt=open("/dev/tty.EV3-SerialPort",O_RDWR);write(bt,start_motor,15);// end connection with EV3close(bt);}
If everything went well you should see the motor starting.
If instead you get an authentication-related error message, download and install the official LEGO app (if you haven’t already), launch it, use it to connect to the EV3 via Bluetooth, check that it really connected, then close it. Somehow that fixes the issue for good. (I know, it’s an ugly hack, but life is short).
Now let’s deconstruct our little script. There are two steps: writing the command and sending the command. Writing the command is the hard part. As you see, it’s not as simple as, say, EV3.start_motor(port = "A", power = 20). Instead of human-readable code what we have here is something called bytecodes. In this particular example every comma-separated piece of the expression inside the inner curly braces is a bytecode - except for the LC1(20) part, which is two bytecodes (more on this in a moment). The first and second bytecodes - 13 and 0 - tell the EV3 the message size (not counting the 13 and the 0 themselves). The third and fourth bytecodes - 0 and 0 - are the message counter.
The fifth bytecode - DIRECT_COMMAND_NO_REPLY - tells the EV3 two things. First, that the instruction is a direct command, as opposed to a system command. Direct commands let you interact with the EV3 and the motors and sensors. System commands let you do things like write to files, create directories, and update the firmware. Second, DIRECT_COMMAND_NO_REPLY tells the EV3 that this is a one-way communication: just start the motor, no need to send any data back. So, the three alternatives to DIRECT_COMMAND_NO_REPLY are SYSTEM_COMMAND_NO_REPLY, DIRECT_COMMAND_REPLY, and SYSTEM_COMMAND_REPLY.
The sixth and seventh bytecodes - 0 and 0 - are, respectively, the number of global and local variables you will need when receiving data from the EV3. Here we’re using a DIRECT_COMMAND_NO_REPLY type of command, so there is no response from the EV3 and hence both bytecodes are zero.
Now we get to the command lui-même. We actually have two commands here, one after the other. The first one, opOUTPUT_POWER, sets how much power to send to the motor. The second one, opOUTPUT_START, starts the motor. Each command is followed by a bunch of local constants (that’s what LC stands for), which contain the necessary arguments. For both commands the first LC0() is zero unless you have multiple EV3 bricks (you can join up to four EV3 bricks together; that’s called a “daisy chain”). Also for both commands, the second LC0() determines the EV3 port. Here we’re using port A - hence LC0(1). Use LC0(2) for port B, LC0(4) for port C, and LC0(8) for port D. Finally, opOUTPUT_POWER takes one additional argument: the desired power. The unit here is percentages: 20 means that we want the motor to run at 20% of its maximum capacity. Unlike the other local constants, this one is of type LC1, not LC0, so it takes up two bytes (see the bytecodes.h file for more on local constants); that is why the message size is 13 even though we only have 12 comma-separated elements.
(Don’t be a sloppy coder like me: instead of having these magic numbers, declare proper variables or constants and use these instead - LC0(port), LC1(power), etc.)
Now let’s send the command we just wrote. On a Mac the way we communicate with other devices via Bluetooth is by writing to (and reading from) tty files that live in the \dev folder (these are not actual files, but file-like objects). If you inspect that folder you will see one tty file for every Bluetooth device you have paired with your computer: your cell phone, your printer, etc. The EV3 file is called tty.EV3-SerialPort. (If you’re curious, here’s all the specs and intricacies of how Bluetooth is implemented on a Mac.)
So, to send the command we wrote before to the EV3 via Bluetooth we open the tty.EV3-SerialPort file (line 16), write the command to it (line 17), and close it (line 20).
That’s it, you can now use C++ to control the EV3 motors.
Just so you know, your command is automatically converted to hexadecimal format before being sent to the EV3 (those LC()s are macros that make the conversion). In other words, your EV3 will not receive {13, 0, 0, 0, DIRECT_COMMAND_NO_REPLY, 0, 0, opOUTPUT_POWER, LC0(0), LC0(1), LC1(20), opOUTPUT_START, LC0(0), LC0(1)}. It will receive \x0D\x00\x00\x00\x80\x00\x00\xA4\x00\x01\x81\x14\xA6\x00\x01 instead. The mapping is provided in the bytecodes.h file. For instance, DIRECT_COMMAND_NO_REPLY is 0x80, opOUTPUT_POWER is 0xA4, and so on.
If you prefer you can hardcode the hexadecimals. This produces the exact same outcome:
#include<unistd.h>
#include<fcntl.h>
#include"ev3sources/lms2012/c_com/source/c_com.h"intmain(){// write command to start motor on port A with power 20charstart_motor[]="\x0D\x00\x00\x00\x80\x00\x00\xA4\x00\x01\x81\x14\xA6\x00\x01";// send command to EV3 via Bluetoothintbt=open("/dev/tty.EV3-SerialPort",O_RDWR);write(bt,start_motor,15);// end connection with EV3close(bt);}
If you master the hexadecimals you can use any language to communicate with the EV3. For instance, in Python you can do this:
# write command to start motor on port A with power 20
start_motor='\x0D\x00\x00\x00\x80\x00\x00\xA4\x00\x01\x81\x14\xA6\x00\x01'+'\n'# send command to EV3 via Bluetooth
withopen('/dev/tty.EV3-SerialPort, mode = 'w+', buffering = 0) as bt:
bt.write(start_motor)
All right then. Now, how do we get data back from the EV3? Well, it’s the reverse process: instead of writing to tty.EV3-SerialPort we read from it. The trick here is to find the sensor data amidst all the other stuff that the EV3 sends back to your computer, but we’ll get there (btw, I’m grateful to the good samaritan who showed me how to do this). To make matters more clear, plug some sensor on port 1 and try running this code:
#include<unistd.h>
#include<fcntl.h>
#include<iostream>
#include"ev3sources/lms2012/c_com/source/c_com.h"intmain(){// read sensor on port 1unsignedconstcharread_sensor[]{11,0,0,0,DIRECT_COMMAND_REPLY,1,0,opINPUT_READ,LC0(0),LC0(0),LC0(0),LC0(0),GV0(0)};// send command to EV3 via Bluetoothintbt=open("/dev/tty.EV3-SerialPort",O_RDWR);write(bt,read_sensor,13);// receive data back from EV3 via Bluetoothunsignedcharsensor_data[255];read(bt,sensor_data,255);for(inti=0;i<255;i++){printf("%x",sensor_data[i]);}// end connection with EV3close(bt);}
The structure of the code is pretty similar to what we had before. The first change is that now our command type is no longer DIRECT_COMMAND_NO_REPLY but DIRECT_COMMAND_REPLY, as we now want to receive data from the EV3. The second change is the sixth bytecode, which is now 1. That means we are now requesting one global variable - we’ll need it to store the sensor data.
The third change is of course the command itself, which is now opINPUT_READ. Its arguments are, in order: the EV3 brick (usually 0, unless you have multiple bricks), the port number (minus 1), the sensor type (0 = don’t change the type), and the sensor mode (0 = don’t change the mode). GV0 is not an argument, but the global variable where the sensor data will be stored. Like the motor power, the data we will get back will be in percentage (alternatively, there is an opINPUT_READSI command that returns the data in SI units).
The fourth change is that we now have a new code block. Its first line - unsigned char sensor_data[255] - creates an array of size 255, with all values initialized to zero. The size is 255 because at this point we don’t know exactly what the actual size of the received data will be, so we want to be safe: the data will be in hexadecimal format, so 255 is about as large as it gets (just as with the data we send, the first two bytes of the data we receive tell us how large the message is - but we can only count up to 255 with two bytes in hexadecimal format, so 255 is the limit here). The second line receives the data and the for loop prints each byte to the screen.
If everything went well you should see as output something like 400021F00000… Try it a couple more times, moving the sensor around in-between. You will notice that the first five digits or so don’t change, and neither do all the others after the sixth or seventh digit. For instance, your results will look like 400023D00000… or 400025B00000… Only two digits or so will change. That is your sensor data! In these three examples, for instance, your data points are 1F, 3D, and 5B. That’s hexadecimal format; in decimal format that means 31, 61, and 91 (here’s a conversion table). Now, once you’ve figured out what the relevant digits are you can get rid of that loop and print only them (say, printf("%x", sensor_data[5]);).
That’s it! Now you can control the motors and read the sensors - that should help you get started. If you inspect the c_com.h files you will see lots of other commands, some of them with usage examples. The way forward is by exploring the firmware code and by trial and error.
Social scientists spend a lot of time writing code that has no physical manifestation. Other than changing pixels in a laptop screen our scripts do not change the material world in any observable way. Meanwhile our colleagues in the engineering department get to build Terminator-like robots and all sorts of fun machines. So I decided I wouldn’t live in perpetual envy anymore.
Enter LEGO Mindstorms.
I’ve been playing experimenting with it for about a week and it’s hard to overstate how much fun it is. Besides lots of regular LEGO pieces you get a mini-CPU, a couple of servomotors, a light sensor, and a proximity sensor. You plug the motors and sensors into the mini-CPU (usually referred to as the “EV3 brick”), use the regular LEGO pieces to assemble whatever robot your imagination produces, and then connect your computer to the EV3 brick to control the motors and read data from the sensors.
EV3 brick, motors, and sensors
The EV3 is the third generation of the LEGO Mindstorms set. The previous ones were called NXT and RCX. The EV3 improves on the NXT on several aspects, including processing power, memory size, number of ports, the inclusion of a microSD card slot, ability to communicate with iOS, and a larger screen. The NXT is still around though, but somehow it costs more than the EV3 ($449.99 vs $349.95 on Amazon); maybe that’s because there are more applications developed for the NXT, as it’s been around since 2006.
The EV3 brick runs on six AA batteries. They go flat really fast (two or three days with moderate use), so I strongly recommend that you buy rechargeable ones. Also, if you don’t want to have to disassemble the robot every time you need to recharge the batteries you should look into this.
Another thing you can do is use a Raspberry Pi instead of the EV3 brick. You can do it yourself if you’re familiar with these things or you can buy a Raspberry Pi that’s already customized for use with the EV3 sensors and motors (see here).
The first step is deciding how exactly you are going to interact with your robot. If you don’t want to write code, LEGO provides an app that lets you program visually, using flowcharts (a bit like in Scratch). You do the programming in your computer and then send the commands to the EV3 brick via USB, Bluetooth, or WiFi. It looks like this:
But more likely you will want to write code (if for no other reason than because the LEGO app is painfully slow - even in my quad-core laptop with 16GB of memory). You have a number of options.
Java
If Java is your cup of tea then leJOS is the way to go about it. leJOS is a Java Virtual Machine that can replace the native firmware of the EV3 brick and let you run Java code on it. It works like this: you get ahold of a microSD card, make it bootable (follow these instructions), insert it into the EV3 brick, and turn it on. If everything went well the EV3 will boot to leJOS (and not to its native firmware). You can then start writing your Java programs, using the leJOS API. (Conveniently, leJOS doesn’t mess with the native firmware: if you remove the microSD card and restart the EV3 it will boot to the native firmware again.)
leJOS initial screen
Python
The Python alternative works similarly to the Java one: you get ahold of a microSD card, make it bootable (following these instructions), insert it into the EV3 brick and turn it on. If everything went well the EV3 will boot the custom debian wheezy OS in the bootable card (and not to its native firmware). You then SSH into the EV3 and use this API to write your programs. (Same as with leJOS, the native firmware is still there for when you want it - just remove the microSD card and restart the EV3.)
The Python alternative sounds great, but somehow I couldn’t make it work. I insert the card, turn the EV3 on, but then nothing (visible) happens, it seems that at some point in the process the EV3 gets stuck. I tried re-flashing the card and also buying a different card, but nothing worked. After two days of failed attempts I gave up (I may come back to it at some point though). (Also note that even if it works the Bluetooth is not stable yet).
Matlab
There are two ways to program the EV3 using Matlab. One is the official EV3 package from MathWorks. Downside: it only works on Windows (32-bit and 64-bit) and on Linux (64-bit). Which takes us to the second alternative: the folks from CyPhy Lab have created a Matlab-EV3 toolkit that should work on Macs as well. Their website is full of documentation and examples to help you get started.
Unlike the Java and Python solutions the Matlab ones do not require any microSD cards. You will boot the EV3 normally and your program will communicate with the native EV3 firmware (via USB, Bluetooth, or WiFi). I think this is a huge plus.
Microsoft API
Microsoft has produced an API that lets you interact with the EV3 from Windows desktop, Windows Phone 8, and WinRT, using .NET, WinJS, and C++. (I don’t want to work with Windows, so I didn’t really look into it.)
C
The EV3’s native firmware is open source and written (largely) in C. If you clone it you can use C to communicate with the EV3. Like the Matlab and Microsoft solutions it doesn’t require any microSD cards; you just turn the EV3 on and communicate with its native firmware via USB, Bluetooth, or WiFi. And unlike the Matlab and Microsoft solutions it’s 100% open source - no need to spend money on Matlab or Windows. The downside is that the firmware source code is only partially documented, so figuring out the right command to do this or that involves a fair amount of trial and error (or asking for help on StackOverflow). But I think it’s still worth it (this is what I’ve been using).
This is it for now. In part 2 I’ll talk a bit about using C to communicate with the EV3. My end goal is to build a self-driving robot, for the fun of it. I’m not sure I’ll use C for everything; I may use it just to communicate with the EV3 and then pass all the data to Python for the machine learning algorithms. We’ll see.
This is something that I’m learning on-the-fly - I’ve never done any serious work in C before and I just learned what bytecode is -, so these posts will not be a tutorial, but merely a way to share these experiences (and perhaps receive some useful feedback from people who actually know this stuff).
I guess that’s the case for many of us who use general-purpose languages (like Python) to do math. When I only knew R and Stata my typical script was a bunch of math expressions, one after the other, some of them put inside functions. And that seemed to work fine. Then I started learning Python and noticed that that’s not what textbook scripts look like. They don’t have functions hanging around by themselves; everything is organized around classes and subclasses.
Realizing the huge gap between the scripts I wrote and the ones on textbooks was a bit demoralizing. Edsger Dijkstra once famously said that “It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration.” I thought that maybe R and Stata were my BASIC, that learning them had destroyed any chances that I would ever write decent code. (Though I have also learned some BASIC, as a kid, so perhaps I was already mentally mutilated by the time I encountered R and Stata).
(This may have impaired my programming skills but I enjoyed tweaking Nibbles. And using goto)
Turns out I made too much of it. Yes, my scripts were ugly and I shouldn’t let things happen outside functions and I shouldn’t have global variables. I firmly intend to go back and refactor my Wordscores and “Fightin’ Words” implementations. But if you are doing math and your script is small and you won’t integrate it into a larger project, then not everything needs be organized around classes.
As someone in the thread above noted, writing math code is fundamentally different than writing a desktop application.
In a program designed to calculate a result, an input gets heavily processed through several phases of computation. It might be useful to define classes here, and it might not: I could see cases where heterogenous input needed to be processed along a common axis, but with the heterogenous identity preserved. Named tuples or classes might serve this goal well. But the concept of the program is essentially a stream or blob of data that moves through a processing pipeline until an intermediate or final result is produced. In a program designed to provide a set of interactive workflows, you can envision the data instead as a set of related data objects – documents perhaps, or account credentials, or card catalog records, or all of the above – upon which the program performs operations at the request of the user. The objects are like a lattice of interrelated data, waiting to be tickled or touched in exactly the right way. Classes fit this paradigm perfectly.
We also don’t get many opportunities to define our own classes when we are doing math. Need a matrix? There is the NumPy array class. Need a time series? There is the pandas Series class. Need a matrix that does relational joins fast? There is the pandas DataFrame class. Math-wise there is a lot of work already done for us. It would be silly to reinvent the wheel. True, it would be great if someone created a sparse matrix class that allowed relational joins (neither scipy.sparse nor pandas.SparseDataFrame do). But defining classes cannot be an end in itself.
Maybe that applies to non-math code as well.
Granted, using a language like Python or Ruby without defining classes may feel weird. It feels like we are not doing actual object-oriented programming. And we are not quite doing functional programming either, since we mutate data, our functions have side-effects, and most of us don’t use lambda calculus. So we are in a sort of programming limbo. Perhaps we should just embrace the functional paradigm wholeheartedly and switch to Haskell or Scheme. Alas, that is hard to do after learning to love Python or Ruby, what with their intuitive syntaxes and extensive libraries, and so many people to answer our questions on StackOverflow. So maybe we should just learn to love the limbo.
This the final part of our Selenium tutorial. Here we will cover headless browsing and parallel webscraping.
why switch to headless browsing
While you are building and testing your Selenium script you want to use a conventional browser like Chrome or Firefox. That way you can see whether the browser is doing what you want it do. You can see it clicking the buttons, filling out the text fields, etc. But once you are ready to release your bot into the wild you should consider switching to a headless browser. Headless browsers are browsers that you don’t see. All the action happens in the background; there are no windows.
Why would anyone want that? Two reasons.
First, you decrease the probability of errors. Your bot will be doing only one thing: interacting with the HTML code behind the website. Opening windows and rendering content can always result in errors, so why do these things now that your script is functional? This is especially the case if you are parallelizing your webscraping. You don’t want five or six simultaneous Chrome windows. That would make it five or six times more likely that you get a graphics-related error.
Second, it makes it easier to use remote computers. Like Amazon EC2, Google Compute Engine, or your university’s supercomputing center. Remote computers have no displays, the standard way to use them is via the command line interface. To use software that has a graphical user interface, like Chrome or Firefox, you need workarounds like X11 forwarding and virtual displays. Since a headless browser doesn’t have a graphical user interface you don’t need to worry about any of that. So why complicate things?
I know, it sounds weird to not see the action. But really, you’ve built and tested your script, you know it works, so now it’s time to get rid of the visual. In the words of Master Kenobi, “stretch out with your feelings”.
using a headless browser
There are several headless browsers, like ZombieJS, EnvJS, and PhantomJS (they tend to be written in JavaScript - hence the JS in the names). PhantomJS is the most popular, so that’s my pick (the more popular the tool the more documented and tested it is and the more people can answer your questions on StackOverflow).
Download PhantomJS and save the binary to the same folder where you downloaded chromedriver before. That binary contains both the browser itself and the Selenium driver, so there is no need to download a “phantomjsdriver” (actually they call it ghostdriver; it used to be a separate thing but now it’s just embedded in the PhantomJS binary, for your convenience).
You can start PhantomJS the same way you start Chrome.
path_to_phantomjs='/Users/yourname/Desktop/phantomjs'# change path as needed
browser=webdriver.PhantomJS(executable_path=path_to_phantomjs)
But you don’t want that. Websites know which browser you are using. And they know that humans use Chrome, Firefox, Safari, etc. But PhantomJS? ZombieJS? Only bots use these. Hence many websites refuse to deal with headless browsers. We want the website to think that we are using something else. To do that we need to change the user-agent string.
dcap=dict(DesiredCapabilities.PHANTOMJS)dcap["phantomjs.page.settings.userAgent"]=("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/53 ""(KHTML, like Gecko) Chrome/15.0.87")
Now we start PhantomJS but using the modified user-agent string.
path_to_phantomjs='/Users/yourname/Desktop/phantomjs'# change path as needed
browser=webdriver.PhantomJS(executable_path=path_to_phantomjs,desired_capabilities=dcap)
And that’s it. The rest of your code doesn’t change. The only difference is that when you run your script you won’t see any windows popping up.
parallel webscraping
If you have tons of searches to do you may launch multiple instances of your code and have them run in parallel. Webscraping is a case of embarrassingly parallel problem, since no communication is required between the tasks. You don’t even need to write a parallel script, you can simply copy and paste your script into n different files and have each do 1/n of the searches. The math is simple: if you divide the work by two you get done in half the time, if you divide the work by 10 you get done in a tenth of the time, and so on.
That is all very tempting and our first instinct is to parallelize as much as possible. We want to launch hundreds or thousands of simultaneous searches and be done in only a fraction of the time. We want to build and deploy a big, scary army of bots.
But you shouldn’t do that, both for your own sake and for the sake of others.
First of all you don’t want to disrupt the website’s operation. Be especially considerate with websites that normally don’t attract a lot of vistors. They don’t have that many servers, so if you overparallelize you can cause the whole thing to shut down. This is really bad karma. Remember that other human beings also need that website - to make a living (if they are the ones running it) or for their own research needs (people like you). Be nice.
Second, if you overparallelize your bots may lose their human façade. Humans can only handle a couple of simultaneous searches, so if the webadmins see 300 simultaneous searches coming from the same IP address they will know that something is off. So even if you don’t bring the website down you can still get in trouble. You worked hard to put your Selenium script together, you don’t want your bots to lose their cover now.
So, proceed gently. How many parallel bots is safe? I have no idea. That depends entirely on the website, on how many IPs you are using, on whether you have proper delays between your searches (see part 4), etc. As a general rule I wouldn’t have more than three or four parallel bots per IP address (because that’s what a human being could manage).
Multiple IPs don’t mean you are safe though. If those IP addresses are all from the same place - say, your campus - then you are not fooling anyone. Suppose your university subscribes to LexisNexis and that, on average, about 500 people use it everyday on campus. Then you deploy 500 parallel bots. The LexisNexis webadmins will see that there is a sudden 100% increase in traffic coming from your campus. It doesn’t matter whether you are using 1 or 500 IP addresses.
If you get caught things can get ugly. LexisNexis, for instance, may cut off your entire university if they catch even one person webscraping. Imagine being the person who caused your entire university to lose access to LexisNexis. Or to Factiva. Or to any other major database on which thousands of people in your campus rely for their own research needs. Bad, bad karma.
By the way, LexisNexis’ terms of use explicitly prohibit webscraping; check item 2.2. It’s safe to have up to 3 or 4 parallel bots; as long as you have proper delays between searches (see part 4) no one can tell your 3-4 bots from a human being. Remember: Selenium doesn’t connect to the site directly but through the browser, so all that LexisNexis sees is Chrome or Firefox or Safari. But if you overparallelize then you may become the person who caused LexisNexis to cut off your entire university.
(A quick rant: item 2.2 is immoral. Universities pay millions of dollars to LexisNexis in subscriptions but then when we really need to use the content they say we can’t? That’s like selling 1000 gallons of water but delivering only one drop at a time. This is especially outrageous now that we have the hardware and software to text-mine millions of texts. The optimal answer here is buyers’ collusion but I don’t see that happening. So I see our webscraping LexisNexis as a legitimate act of civil disobedience.)
In sum, parallelizing can speed things up but you need to be careful and mindful of others when doing it.
This is it. I hope you have found this tutorial useful. There is a lot I haven’t covered, like interacting with hidden elements and executing arbitrary JavaScript code, but at this point you can figure out anything by yourself.
I wrote this tutorial because even though Selenium is perhaps the most powerful webscraping tool ever made it’s still somewhat unknown to webscrapers. If you found it useful as well then help spread the word.
