Story: The Morning After Valentine’s Day

When I finally open my eyes and look at the clock, it is 8am. It doesn’t feel like it’s only been eight hours, though. I’ve just had a long and complicated dream that I can’t remember much of anymore, except that I think I was running a lot, and trying to not die, so I somehow feel sore.

That NyQuil stuff really works, I think to myself, and crawl out of bed. (Even though it’s like trying to drink mouthwash.) I haven’t slept that soundly all week. Or maybe I’m finally slowly recovering from my cold, and that’s why that night was better? All I know is that I’m glad I didn’t spend another night coughing my lungs out and struggling to get some shut-eye.

I drag my sorry butt out of bed and head over to my nearby computer. It’s the Friday morning before Harvard-MIT math tournament, which means that the server is getting more traffic than usual, and I was supposed to have upgraded the server in anticipation yesterday night. But Valentine’s Day was too hectic for me this year, and I never got around to it.

Not hectic for any romantic reasons. It was because I had 5.5 hours of class more or less consecutively, after which I rushed back to my place to teach for another four hours straight, all the while coughing like a banshee. Okay, so maybe I would have slept fine without the NyQuil.


Officially, this is supposed to be the software team’s job. But the HMMT website has become a complete mess that I think I might be the only person left that still knows more than half of what it’s doing. (Well, actually, Banana seems have figured out a lot of it too.) It is sort the equivalent of Frankenstein’s monster, with parts being sewn in and out over the last who-knows-how-many-years by random undergraduates with various degrees of competence, and held together by the seams with spit and prayers. The top of the main settings files still reads

Django settings for mysite project.
For more information on this file,
see https://docs.djangoproject.com/en/1.6/topics/settings/
For the full list of settings and their values,
see https://docs.djangoproject.com/en/1.6/ref/settings/

where the “1.6” version number still makes me wince every time I see it (that means this file was created before I made the IMO). I am looking forward to the end of this tournament season so I can burn the whole website to the ground and re-write it.

(Making matters worse, in terms of “various degrees of competence”, I am on the low end, with no formal CS experience at all. Not good I am in charge.)

So it’s time to bump up the servers belatedly. I need to bump the web server and then the database up from t2.micro to m1.medium. There will be some downtime, but no big deal — everyone’s probably still asleep. This should only take a few minutes, and then I can work on getting the grader ready for the approximately 100,000 grading inputs that we’re going to force-feed it on Saturday.

So I push a few buttons, and let Amazon Web Services do its magic, just like the last five times I had to do this.


Something’s wrong. It’s been ten minutes already, and the website still won’t load. The upgrades should be done by now. I refresh again, and realize that it’s throwing a 500 error.

I feel a twinge of despair, which causes my cough to start to return. Looks like the NyQuil wore off. It always starts like this. One little error, followed by another, and then… I clench my teeth and SSH into the new server, and navigate to the log files (which I remember having to set up myself a few years ago, precisely for situations like this).

And indeed, when I get there, the same message is repeated, over and over, for the last several minutes, like a harbinger of doom:

OperationalError: (1045, "Access denied for user 'ebroot'@'xx.xx.xx.xx' (using password: YES)")

Oh, well, here we go.


My first guess is VPC groups, which have bitten me plenty in the past. Unfortunately, tinkering with these has no effect, so it seems like there is something new going on.

I double-check and triple-check the password, but it seems right. (This is the same error you get if you enter the wrong password!) So that’s not it, either.

This is a connection issue, so, the first thing I want to do is figure out whether the issue is with the upgraded server or the upgraded database. So, I go to my command bar and launch MySQL workbench, which promptly gives me

mysql-workbench: command not found

Oh, right, I don’t have Workbench installed on my desktop. So I go over to my laptop, which does have it, and try to fire up a connection to the database. This time, it gives me

喊叫 org.freedesktop.secrets 發生錯誤

Uhh, what? Okay, I have a keyring error of some sort here. I decide it’s not worth it to try to fix the workbench, and decide to do it the old fashioned way. I fire another terminal and call mysql, which promptly returns with:

mysql: command not found

Great. Okay, well, I guess I can install it, no problem. So I type in sudo pacman -S mysql which gives

:: 有 2 個提供者可供 mysql:

:: 軟體庫 extra

1) mariadb

:: 軟體庫 community

2) percona-server

輸入某個數字(預設=1):

Huh? Oh, Arch Linux prefers MariaDB over MySQL, I remember now. That means SQL, unlike MariaDB, is not pre-packaged, and I’ll need to download it from the Arch User Repository and compile it form source. So I download the PKGBUILD and let it start going.

Unfortunately, this means that the binary needs to build from source, and so maybe minutes later I’m staring at the build progress and it’s at 14%. All the while the website is down, and people are starting to notice. I can feel the tension all over my body as I realize that the tournament will simply not function if I can’t get this back up and working, and the nightmare from February 2018 starts coming back to my mind again. I explode into a symphony of coughs as I struggle to gain my composure.

The compile-from-source doesn’t respond to my please. I decided that’s not good enough, and I have to do something faster.


At this point I decide I maybe should get help, so I hit up Banana who has saved HMMT on numerous occasions as well, and ask him why I can’t connect to the HMMT database. He’s done so successfully on his computer in the past, and he even has a little script he’s written to do so. His response puzzles me:

“Seems connectable. Might need to reset your DNS?”

Uh, what?

This is the only lead I have to go after, so I start prodding him like crazy, since I can’t seem to get it to do anything from my end. At first I get a couple suggestions, which don’t work, but eventually Banana finally gives me:

“Uhhhh. Can you try running db-connect on your machine and going from there? I have to start hauling.”

Right. Unlike me, Banana is actually helping the directors move around the 30 gallons of apple juice and 50 gallons of mango nectar and way-too-many gallons of water that will be used today or tomorrow. So I am really on my own.


At least I now know that it’s on the database end, sorta. I quickly locate db-connect: it’s the mini-script that Banana has been using to do the connection. Maybe this will let me get in and see what’s happening. I type:

./db-connect.sh dev

which promptly gives

mysql: command not found

Ah right, we’re still working on that, huh? The sql compilation is going nowhere fast, and so I have to think of something else.

I think of one possible approach: the workbench won’t work on my laptop, but maybe it’ll work on my desktop? I order pacman to install MySQL Workbench on my desktop too, and after a couple dozen agonizing seconds, the download is all done. To my delight, there is no error about org.freedesktop.secrets, and so I impatiently set everything up and login to find:

ERROR 1045 (28000):
Access denied for user 'ebroot'@'xx.xx.xx.xx' (using password: YES)

Oh, no, no, no. I explode into another fit of coughs which prevent me from screaming at the monitor in frustration.


At this point, I decide this isn’t worth fighting. I can figure it out another time. For now, the show must go on.

That means that if I restore a backup of the old database — reverting it back to how it was yesterday, when everything went totally fine — then I can at least get the computer to work now, and worry about what the error was later.

Unfortunately, this is a painfully slow process. (The way backups work is that they don’t replace the existing database. Instead, it creates a brand new database somewhere on the cloud, but that has a copy of the same data as the point in time.) I load the backup, and twitch in agony as it slowly creates a new database from the image, setting everything back to how it was earlier. I hope that’s good enough.

After what feels like an eternity, the database is all set. I change the pointer of my now-working WorkBench to the new database and try to connect to see what happens, only to be greeted with 45 seconds of nothing, followed by a simple error message telling me that the connection failed.

Why? Oh yeah, I didn’t send the VPC for the new database. I do that, exhaling, it should be fine now, and connect to the new restored database, only to find:

ERROR 1045 (28000):
Access denied for user 'ebroot'@'xx.xx.xx.xx' (using password: YES)

I practically choke on my own spit, which results in another several seconds of me wheezing like heck.

Nooooooooooooooooooooooo.

Okay, I have to fix this now.


The only clue I have is that the database script from Banana still works. But on my computer I can’t for some reason run it.

I go back to Google (which I have been using extensively the whole time), and then after another few minutes of frantic searching, realize that MariaDB is actually good enough for me: once I have that installed, I’ll have a (slightly different) SQL client, but the script should work. Hmm.

Since MariaDB is pre-packaged, that means the installation is easy, and I run it. This might be it. I fire the script again, and run into:

mysql: unknown option '--enable-cleartext-plugin'

Oh, huh. Okay, well, maybe it doesn’t matter. I delete that flag from the script, and get

mysql: unknown variable 'ssl-mode=VERIFY_IDENTITY'

Uhh. Let’s cross our fingers that doesn’t matter either? I try that again, and — much to my amazement — the connection works.

I start examining the db-connect script closely, and see that instead of the user ebroot it’s connecting using some user name dev. So maybe there is some new permission issues with ebroot? With my new connection, I try to

GRANT ALL PRIVILEGES ON *.* TO 'ebroot'@'%';

which then returns with the following message:

ERROR 1045 (28000):
Access denied for user 'dev'@'%' (using password: YES)

Uhhh.

Okay, maybe something else. I read the db-connect script again. Is there anything else that’s different? Well, there’s one more change in the code that I can at least work with: there is a CA certificate that’s being used.

I fire up WorkBench again, and try to log in again, but this time I pass a newfound CA certificate (appropriately named rds-combined-ca-bundle.pem) and to my relief, I find that I can now log in as ebroot. That’s the issue!


Except I have no idea how to make Django do that and I have no intention if finding out. But another Google search suggests the answer: now that I’m finally connected with ebroot I type

ALTER USER 'ebroot'@'%' REQUIRE NONE;

And breathed a sigh of relief when I refreshed hmmt.co, and harmony was restored in the world.

I looked at the clock. It was 11am. There goes my whole morning. I go downstairs to drink a bottle of Soylent for breakfast, resolving to never become a programmer for a living, or at least to get some proper training first if I ever consider it.

Meanwhile, the rest of the Harvard-MIT math tournament go on with their day, blissfully unaware of the debacle narrowly averted.

Advertisements

SysRq on Arch Linux Mac Mini

This post documents my adventures of getting the SysRQ key working on my Mac Mini and Macbook (both running Arch Linux). The suggestions of loadkeys and keyfuzz that are the first search entries don’t work for me, so some more sophisticated black magic was necessary.

Remapping the Fn keys

This step is technically optional, but I did it because the function keys are a pain anyways. Normally on Apple keyboards one needs to use the Fn key to get the normal Fn keys to behave as a F<n> keystroke. I prefer to reverse this behavior, so that the SysRq combinations is Alt+F13+F rather than Fn+Alt+F13+F, say.

For this, the advice on the Arch Wiki worked, although it is not thorough on some points that I think should’ve been said. On newer kernels, one does this by creating the file /etc/modprobe.d/hid_apple.conf and writing

options hid_apple fnmode=2

Then I edited the file /etc/mkinitcpio.conf to include the new file:

...
BINARIES=""

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES="/etc/modprobe.d/hid_apple.conf"

# HOOKS
...

Finally, recompile the kernel for this change to take effect. On Arch Linux one can just do this by issuing the command

$ sudo pacman -S linux

which will reinstall the entire kernel.

Obtaining the keystroke

Next, I needed to get the scancode of the key I wanted to turn into the SysRQ key. For me attempting showkey -s did not work so I instead had to use evtest, as described in this Arch Wiki.

$ sudo pacman -S evtest
$ sudo evtest
No device specified, trying to scan all of /dev/input/event*
Available devices:
/dev/input/event0:  Logitech USB Receiver
/dev/input/event1:  Logitech USB Receiver
/dev/input/event2:  Apple, Inc Apple Keyboard
/dev/input/event3:  Apple, Inc Apple Keyboard
/dev/input/event4:  Apple Computer, Inc. IR Receiver
/dev/input/event5:  HDA NVidia Headphone
/dev/input/event6:  HDA NVidia HDMI/DP,pcm=3
/dev/input/event7:  Power Button
/dev/input/event8:  Sleep Button
/dev/input/event9:  Power Button
/dev/input/event10: Video Bus
/dev/input/event11: PC Speaker
/dev/input/event12: HDA NVidia HDMI/DP,pcm=7
/dev/input/event13: HDA NVidia HDMI/DP,pcm=8
Select the device event number [0-13]: 2
Input driver version is 1.0.1
Input device ID: bus 0x3 vendor 0x5ac product 0x220 version 0x111
Input device name: "Apple, Inc Apple Keyboard"

This is on my Mac Mini; the list of devices looks different on my laptop. After this pressing the desired key yields something which looked like

Event: time 1456870457.844237, -------------- SYN_REPORT ------------
Event: time 1456870457.924097, type 4 (EV_MSC), code 4 (MSC_SCAN), value 70068
Event: time 1456870457.924097, type 1 (EV_KEY), code 183 (KEY_F13), value 1

This is the F13 key which I want to map into a SysRq — the keycode 70068 above (which is in fact a hex code) is the one I wanted.

Using udev

Now that I had the scancode, I cd’ed to /etc/udev/hwdb.d and added a file
90-keyboard-sysrq.hwdb with the content

evdev:input:b0003*
  KEYBOARDKEY_70068=sysrq

One then updates hwdb.bin by running the command

$ sudo udevadm hwdb --update
$ sudo udevadm trigger

The latter command makes the changes take effect immediately. You should be able to test this by running sudo evtest again; evtest should now report the new keycode (but the same scancode).

One can test the SysRQ key by running Alt+SysRq+H, and then checking the dmesg output to see if anything happened:

$ dmesg | tail -n 1
[  283.001240] sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) ...

Enable SysRq

It remains to actually enable SysRQ, according to the bitmask described here. My system default was apparently 16:

$ sysctl kernel.sysrq
kernel.sysrq = 16

For my purposes, I then edited /etc/sysctl.d/99-sysctl.conf and added the line

kernel.sysrq=254

This gave me everything except the nicing of real-time tasks. Of course the choice of value here is just personal preference.

Personally, my main use for this is killing Chromium, which has a bad habit of freezing up my computer (especially if Firefox is open too). I remedy the situation by repeatedly running Alt+SysRq+F to kill off the memory hogs. If this doesn’t work, just Alt+SysRq+K kills off all the processes in the current TTY.

DNSCrypt Setup with PDNSD

Here are notes for setting up DNSCrypt on Arch Linux, using pdnsd as a DNS cache, assuming the use of NetworkManager. I needed it one day since the network I was using blocked traffic to external DNS servers (parental controls), and the DNS server provided had an outdated entry for hmmt.co. (My dad then pointed out to me I could have just hard-coded the necessary IP address in /etc/hosts, oops.)

For the whole process, useful commands to test with are:

  • nslookup hmmt.co will tell you the IP used and the server from which it came.
  • dig http://www.hmmt.co gives much more detailed information to this effect. (From bind-tools.)
  • dig @127.0.0.1 http://www.hmmt.co lets you query a specific DNS server (in this case 127.0.0.1).
  • drill @127.0.0.1 http://www.hmmt.co behaves similarly.

First, pacman -S pdnsd dnscrypt-proxy (with sudo ostensibly, but I’ll leave that out here and henceforth).

Run systemctl edit dnscrypt-proxy.socket and fill in override.conf with

[Socket]
ListenStream=
ListenDatagram=
ListenStream=127.0.0.1:40
ListenDatagram=127.0.0.1:40

Optionally, one can also specify which server which DNS serve to use with systemctl edit dnscrypt-proxy.service. For example for cs-uswest I write

[Service]
ExecStart=
ExecStart=/usr/bin/dnscrypt-proxy \
      -R cs-uswest

The empty ExecStart= is necessary, since otherwise systemctl will complain about multiple ExecStart commands.

This thus configures dnscrypt-proxy to listen on 127.0.0.1, port 40.

Now we configure pdnsd to listen on port 53 (default) for cache, and relay cache misses to dnscrypt-proxy. This is accomplished by using the following for /etc/pdnsd.conf:

global {
    perm_cache = 1024;
    cache_dir = "/var/cache/pdnsd";
    run_as = "pdnsd";
    server_ip = 127.0.0.1;
    status_ctl = on;
    query_method = udp_tcp;
    min_ttl = 15m;       # Retain cached entries at least 15 minutes.
    max_ttl = 1w;        # One week.
    timeout = 10;        # Global timeout option (10 seconds).
    neg_domain_pol = on;
    udpbufsize = 1024;   # Upper limit on the size of UDP messages.
}

server {
    label = "dnscrypt-proxy";
    ip = 127.0.0.1;
    port = 40;
    timeout = 4;
    proxy_only = on;
}

source {
    owner = localhost;
    file = "/etc/hosts";
}

Now it remains to change the DNS server from whatever default is used into 127.0.0.1. For NetworkManager users, it is necessary to edit /etc/NetworkManager/NetworkManager.conf to prevent it from overriding this file:

[main]
...
dns=none

This will cause resolv.conf to be written as an empty file by NetworkManager: in this case, the default 127.0.0.1 is used as the nameserver, which is what we want.

Needless to say, one finishes with

systemctl enable dnscrypt-proxy
systemctl start dnscrypt-proxy
systemctl enable pdnsd
systemctl start pdnsd

Arch Linux on a Mac Mini

This post briefly outlines the process of setting up a dual boot OSX and Arch Linux on a Mac Mini. This is mostly for my reference in the likely event that I will be doing anything similar in some years, so it assumes some competence; fortunately, the Arch Wiki’s Beginner’s Guide probably fills in any gaps I left out. Obligatory Disclaimer: Use at your own risk or not at all.

This is almost the same as any other installation of Arch Linux, with a few changes that took some hours of frustration to figure out because of the EFI booter. My method is to create the partitions in Disk Utility, install rEFInd, and then install the grub bootloader into /dev/sda1.

Setup done in OSX

  1. First, install rEFInd. This worked out of the box for me, and makes it possible to boot via USB.
  2. Create a Arch Linux installer USB by dd-ing (or anything else) the latest installation medium onto a USB drive.
  3. Set up the partitions; I find it easier (and less dangerous) to just use the OSX Disk Utility to do this. See, for example, the procedure here. My OSX installations appear to come with three partitions, a small one called “EFI”, a main “OS X HD” partition, and then a small “Recovery HD”, like so:
    NAME   LABEL       TYPE   SIZE
    sda                disk 931.5G 
    |-sda1 EFI         part   200M
    |-sda2 OS X HD     part 927.9G
    `-sda3 Recovery HD part 619.9M

    (This output is from lsblk, and is not what Disk Utility looks like).
    I like to create a partition for my Arch Linux system (which I name “Arch”) and a fifth partition just for the /home directory (which I name Home”). This leaves me to something like

    NAME   LABEL       TYPE   SIZE
    sda                disk 931.5G
    |-sda1 EFI         part   200M
    |-sda2 OS X HD     part 476.9G
    |-sda3 Recovery HD part 619.9M
    |-sda4 Arch        part   179G
    `-sda5 Home        part   272G

Booting into the USB and finishing up the partitions

Now that the partitions and rEFInd is set up, and the USB is written, we can proceed with the actual installation.
At this point, one can basically follow the standard procedure with a few changes.

  1. Reboot the device into the USB. Since rEFInd is installed, it should give you the option of booting into the USB.
  2. Establish an Internet as required.
  3. We’ve already created the partitions in Disk Utility above, so there is no need to change the partitions themselves now. However, it is necessary to format the newcly created partitions above. In my case, the relevant commands are
    # mkfs.ext4 /dev/sda4
    # mkfs.ext4 /dev/sda5
    

    Warning: Please, please make sure that you are formatting the right partitions. The command lsblk -f will print out the partitions and their labels.

  4. Now we need to mount the directories. There are three directories we need to mount, the main filesystem and the home directory, as well as the EFI boot directory. The part that was non-obvious to me is that the boot directory we want is actually the “EFI” directory (likely /dev/sda1) that OSX already provides. The relevant commands in my case were
    # mount /dev/sda4 /mnt
    # mkdir /mnt/home
    # mount /dev/sda5 /mnt/home
    # mkdir /mnt/boot
    # mount /dev/sda1 /mnt/boot
    
  5. Now you can happily install the base system and generate an fstab file:
    # genfstab -U -p /mnt >> /mnt/etc/fstab
    

Configuring the base system and installing the bootloader

  1. Now we can chroot into the system and follow all the directions, up to (but not including) installing the bootloader.
  2. I could not get gummiboot to work but maybe you will have better luck. Fortunately, with /dev/sda1/ mounted as /boot, I got GRUB to work nicely.
    # pacman -S grub
    # grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=arch_grub --recheck
    # grub-mkconfig -o /boot/grub/grub.cfg
    
  3. Now we can exit the chroot environment and power down the system.

If all goes well, upon rebooting, rEFInd will now boot into the complete Arch Linux system.

PDF Compression

I always scan copies of letters into my computer before I send them out. So I had a bunch of large PDF’s sitting around hogging my Dropbox space.

One day I found this blog post which claimed that simply running (in Bash) the commands
$ pdf2ps original.pdf temp.ps
$ ps2pdf temp.ps new.pdf

would decrease the file size. (The two commands are part of GhostScript, which I had installed on my Linux boxes anyways.) I couldn’t resist trying it — and miraculously, it worked. It generally decreases my scans by a factor of 10 (from 20MB to 2MB or so).

I have no clue why this works, although it probably has something to do with the fact that the PDF’s are scanned pages . Anyone care to enlighten me?