https://evanchen.cc/upload/MOHS-hardness.pdf

In short, the scale runs from 0M to 50M in increments of 5M, and every USAMO / IMO problem on my archive now has a rating too.

My hope is that this can be useful in a couple ways. One is that I hope it’s a nice reference for students, so that they can better make choices about what practice problems would be most useful for them to work on. The other is that the hardness scale contains a very long discussion about how I judge the difficulty of problems. While this is my own personal opinion, obviously, I hope it might still be useful for coaches or at least interesting to read about.

As long as I’m here, I should express some concern that it’s possible this document does more harm than good, too. (I held off on posting this for a few months, but eventually decided to at least try it and see for myself, and just learn from it if it turns out to be a mistake.) I think there’s something special about solving your first IMO problem or USAMO problem or whatever and suddenly realizing that these problems are actually doable — I hope it would not be diminished by me rating the problem as 0M. Maybe more information isn’t always a good thing!

]]>Math must be presented for System 1 to absorb and only incidentally for System 2 to verify.

I finally have a sort-of formalizable guideline for teaching and writing math, and what it means to “understand” math. I’ve been unconsciously following this for years and only now managed to write down explicitly what it is that I’ve been doing.

(This post is written from a math-centric perspective, because that’s the domain where my concrete object-level examples from. But I suspect much of it applies to communicating hard ideas in general.)

The quote above refers to the System 1 and System 2 framework from *Thinking, Fast and Slow*. Roughly it divides the brain’s thoughts into two categories:

- S1 is the part of the brain characterized by fast, intuitive, automatic, instinctive, emotional responses, For example, when you read the text “2+2=?”, S1 tells you (without any effort) that this equals 4.
- S2 is the part of the brain characterized by slow, deliberative, effortful, logical responses; for example, S2 is used to count the number of words in this sentence.

(The link above gives some more examples.)

The premise of this post is that understanding math well is largely about having the concept resonate with your S1, rather than your S2. For example, let’s take groups from abstract algebra. Then I claim that

is a group under the usual multiplication. Now, if you have a student who’s learning group theory for the first time, the only way they could see this is a group is to compare it against a list of the group axioms, and have their S2 verify them one by one. But experienced people don’t do this: their S1 automatically tells them that “feels” like a group (because e.g. it’s closed and doesn’t have division-by-zero issues).

I think this S1-level understanding is what it means to “get it”. Verifying a solution to a hard olympiad problem by having S2 check each individual step is straightforward in principle, albeit time-consuming. The tricky part is to get this solution to resonate with S1. Hence my advice to never read a solution line by line.

What this means is that if you’re trying to teach someone an idea, then you should be focusing on trying to get their S1 to grasp it, rather than just their S2. For example, in math it’s not enough to just give a sequence of logical steps which implies the result: *give it life*.

Here are some examples of ways I (try to) do this.

First, **giving good concrete examples**. S1 reacts well when it “sees” a concrete object like above, and can see some intuitive properties about it right away. Abstract “symbol-pushing” is usually left to S2 instead.

Similarly, **drawing pictures**, so your S1 can *actually* see the object. On one extreme end, you can *write* something like “a point $S$ lies on the polar of $T$ if and only if $T$ lies on the polar of $S$”, but it’s much better to just have a picture:

You can even do this for things that aren’t really geometrical in nature. For example, my Napkin features the following picture of cardinal collapse when forcing.

Third, **write like you talk**, and share your feelings. S1 is emotional. S1 wants to know that compactness is a *good* property for a space to have, or that non-Noetherian rings are *way too big* and “only weirdos care about non-Noetherian rings” (just kidding!), or that ramified primes are the “finitely many edge cases” and aren’t worth worrying about. These S1 reactions you get are the things you want to pass on. In particular, avoid standard formal college-textbook-bleed-your-eyes-dry-in-boredom style. (To be fair, not all textbooks do this; this is one reason why I like Pugh’s book so much, for example.)

Even the mechanics on the page can be made to accommodate S1 in this way. S1 can’t read a wall of text; S2 has to put in effort to do that. But S1 can pick out section headers, or **bolded phrases like this one**, and so on and so forth. That’s why in Napkin all the examples are in separate red boxes and all the big theorems are in blue boxes, and important philosophical points are typeset in bold centered green text. This way S1 naturally puts its attention there.

On the flip side, if you’re trying to *learn* something, there’s a common failure mode where you try to keep forcing S2 to do something unnatural (rather than trying to have S1 figure it out). This is the kind of thing when you don’t understand what the Chinese Remainder Theorem is trying to say, so you try to fix this by repeatedly reading the proof line by line, and still not really understanding what is going on. Usually this ends up in S2 getting tired and not actually reading the proof after the third or fourth iteration.

(For the Chinese remainder theorem the right thing to do is ask yourself why any arithmetic progression with common difference 7 must contain multiples of 3: credits to Dominic Yeo again for that. I’m not actually sure what you’re supposed to do when stuck on math in general. Usually I just ask my friends what is going on, or give up for now and come back later.)

Actually, I really like the advice that SSC mentions: “develop instincts, then use them”.

]]>Because we sure do an *awful* job of being supportive of the students, or, well, really doing anything at all. There’s no practice material, no encouragement, or actually no form of contact whatsoever. Just three unreasonably hard problems each month, followed by a score report about a week later, starting in December and dragging in to April.

One of a teacher’s important jobs is to encourage their students. And even though we get the best students in the USA, probably we shouldn’t skip that step entirely, especially given the level of competition we put the students through.

So, what should we do about it? Suggestions welcome.

]]>This year’s USA delegation consisted of leader Po-Shen Loh and deputy leader Yang Liu. The USA scored 227 points, tying for first place with China. For context, that is missing a total of four problems across all students, which is actually kind of insane. All six students got gold medals, and two have perfect scores.

- Vincent Huang 7 7 3 7 7 7
- Luke Robitaille 7 6 2 7 7 6
- Colin Shanmo Tang 7 7 7 7 7 7
- Edward Wan 7 6 0 7 7 7
- Brandon Wang 7 7 7 7 7 1
- Daniel Zhu 7 7 7 7 7 7

Korea was 3rd place with 226 points, just one point shy of first, but way ahead of the 4th place score (with 187 points). (I would actually have been happier if Korea had tied with USA/China too; a three-way tie would have been a great story to tell.)

You can find problems and my solutions on my website already, and this year’s organizers were kind of enough to post already the official solutions from the Problem Selection Committee. So what follows are merely my opinions on the problems, and my thoughts on them.

First, comments on the individual problems. (Warning: spoilers follow.)

- This is a standard functional equation, which is quite routine for students with experience. In general, I don’t really like to put functional equations as opening problems to exams like the IMO or USAJMO, since students who have not seen a functional equation often have a difficult time understanding the problem statement.
- This is the first medium geometry problem that the IMO has featured since 2012 (the year before the so-called “Geoff rule” arose). I think it’s genuinely quite tricky to do using only vanilla synthetic methods, like the first official solution. In particular, the angle chasing solution was a big surprise to me because my approach (and many other approaches) start by
*eliminating*the points and from the picture, while the first official solution relies on them deeply. (For example one mightt add and and noting and are cyclic so it is equivalent to prove lies on the radical axis of and ). That said, I found that the problem succumbs to barycentric coordinates and I will be adding it as a great example to my bary handout. The USA students seem to have preferred to use moving points, or Menelaus theorem (which in this case was just clumsier bary). - I actually felt the main difficulty of the problem was dealing with the artificial condition. Basically, the problem is about performing an operation while trying to not disconnect the graph. However, this “connectedness” condition, together with a few other necessary weak hypotheses (namely: not a clique, and has at least one odd-degree vertex) are lumped together in a misleading way, by specifying 1010 vertices of degree 1009 and 1009 vertices of degree 1010. This misleads contestants into, say, splitting the graph into the even and odd vertices, while hiding the true nature of the problem. I do think the idea behind the problem is quite cute though, despite being disappointed at how it was phrased. Yang Liu suggested to me this might have been better off at the IOI, where one might ask a contestant to output a sequence of moves reducing it to a tree (or assert none exist).
- I liked the problem (and I found the connection to group theory amusing), though I think it is pretty technical for an IMO 1/4. Definitely on the hard side for inexperienced countries.
- This problem was forwarded to the USAMO chair and then IMO at my suggestion, so I was very happy to see it on the exam. I think it’s a natural problem statement that turns out to have an unexpecetdly nice solution. (And there is actually a natural interpretation of the statement via a Turing machine.) However, I thought it was quite easy for the P5 position (even easier than IMO 2008/5, say).
- A geometry problem from one of my past students Anant Mudgal. Yang and I both solved it very quickly with complex numbers, so it was a big surprise to us that none of the USA students did. I think this problem is difficult but not “killer”; easier than last year’s IMO 2018/6 but harder than IMO 2015/3.

For gold-level contestants, I think this was the easiest exam to sweep in quite a few years, and I confess during the IMO to wondering if we had a small chance at getting a full 252 (until I found out that the marking scheme deducted a point on P2). Problem 2 is tricky but bary-able, and Problem 5 is quite easy. Furthermore neither Problem 3 or Problem 6 are “killers” (the type of problem that gets fewer than 20 solves, say). So a very strong contestant would really have 3 hours each day to work on a Problem 3 or Problem 6 which is not too nightmarish. I was actually worried for a while that the gold cutoff might be as high as 34 points (five problems), but I was just being silly.

]]>People sometimes ask me, why do we have international students at MOP? Doesn’t that mean we’re training teams from other countries? So I want to make this clear now: the purpose of MOP is not to train and select future IMO teams.

I know it might seem that way, because we invite by score and grade. But I really think the purpose of MOP is to give each one of you the experience of working hard and meeting new people, among other things. Learn math, face challenges, make friends, the usual good stuff, right? And that’s something you can get no matter what your final rank is, or whether you make IMO or EGMO or even next year’s MOP. The MOP community is an extended family, and you are all part of it now.

What I mean to say is, the camp is designed with all 80 of you in mind. It made me sad back in 2012 when one of my friends realized he had little chance of making it back next year, and told me that MAA shouldn’t have invited him to begin with. Even if I can only take six students to the IMO each year, I never forget the other 74 of you are part of MOP too.

This means one important thing: everyone who puts in their best shot deserves to be here. (And unfortunately this also means there are many other people who deserve to be here tonight too, and are not. Maybe they solved one or two fewer problems than you did; or maybe they even solved the same number of problems, but they are in 11th grade and you are in 10th grade.)

Therefore, I hope to see all of you put in your best effort. And I should say this is not easy to do, because MOP is brutal in many ways. The classes are mandatory, we have a 4.5-hour test every two days, and you will be constantly graded. You will likely miss problems that others claim are easy. You might find out you know less than you thought you did, and this can be discouraging. Especially in the last week, when we run the TSTST, many of you will suddenly realize just how strong Team USA is.

So I want to tell you now, stay determined in the face of adversity. This struggle is your own, and we promise it’s worth it, no matter the outcome. We are rooting for you, and your friends sitting around you are too. (And if the people around you aren’t your friends yet, change that asap.)

]]>
As usual, to make these things easier to talk about, I’m going to introduce some words to describe these two. Taking a page from martial arts, I’m going to run with **hard** and **soft** techniques.

A **hard** technique is something you try in the hopes it will prove something — ideally, solve the problem, but at least give you some intermediate lemma. Perhaps a better definition is “things that will end up in the actual proof”. Examples include:

- Angle chasing in geometry, or proving quadrilaterals are cyclic.
- Throwing complex numbers at a geometry problem.
- Plugging in some values into a functional equation (which gives more equations to work with).
- Taking a given Diophantine equation modulo to get some information, or taking -adic evaluations.
- Trying to perform an induction, for example by deleting an element.
- Trying to write down an inequality that when summed cyclically gives the desired conclusion.
- Reducing the problem to one or more equivalent claims.

and so on. I’m sure you can come up with more examples.

In contrast, a **soft** technique is something you might try to help you understand the problem better — even if it might not prove anything. Perhaps a better definition is “things not written up”. Examples include:

- Examining particular small cases of the problem.
- Looking at the equality cases of a min/max problem.
- Considering variants of the problem (for example, adding or deleting conditions).
- Coming up with lots of concrete examples and playing with them.
- Trying to come with a counterexample to the problem’s assertion and seeing what the obstructions are.
- Drawing pictures, even on non-geometry problems (see JMO2 and JMO5 in my 2019 notes for example).
- Deciding whether or not a geometry problem is “purely projective”.
- Counting the algebraic degrees of freedom in a geometry problem.
- Checking all the linear/polynomial solutions to a functional equation, in order to get a guess what the answer might be.
- Blindly trying to guess solutions to an algebraic equation.
- Making up an artificial unnatural function in a functional equation, and then trying to see why it doesn’t work (or occasionally being surprised that it does work).
- Thinking about why a certain hard technique you tried failed, or even better convincing yourself it cannot work (for example, this Diophantine equation has a solution modulo every prime, so stop trying to one-shot by mods).
- Giving a heuristic argument that some claim should be true or false (“probably is odd infinitely often”), or even easy/hard to prove.

and so on. There is some grey area between these two, some of the examples above might be argued to be in the other category (especially in context of specific problems), but hopefully this gives you a sense of what I’m talking about.

If you look at things I wrote back when I was in high school, you’ll see this referred to as “attacking” and “scouting” instead. This is too silly for me now even by my standards, but back then it was because I played a lot of *StarCraft: Brood War* (I’ve since switched to StarCraft II). The analogy there is pretty self-explanatory: knowing what your opponent is doing is important because your army composition and gameplay decisions should change in reaction to more information.

Now after all that blabber, here’s the action item for you all: **you should try soft techniques when stuck**.

When you first start doing a problem, you will often have some good ideas for what to try. (For example: a wild geometry appeared, let’s scout for cyclic quadrilaterals.) Sometimes if you are lucky enough (especially if the problem is easier) this will be enough to topple the problem, and you can move on. But more often what happens is that eventually you run out of steam, and the problem is still standing. When that happens, my advice is to try doing some soft techniques if you haven’t already done so.

Here’s an example that I like to give.

**Example 1** **(USA TST 2009)**

Find all real numbers , , which satisfy

A common first thing that people will try to do is add the first two equations, since that will cause the terms to cancel. This gives a factor of in the left and an in the right, so then maybe you try to submit that into the in the last equation, so you get , cool, there’s no more linear terms. Then. . .

Usually this doesn’t end well. You add this and subtract that and in the end all you see is equation after equation, and after a while you realize you’re not getting anywhere.

So we’re stuck now. What to do? I’ll now bring in two of the soft techniques I mentioned earlier:

- Let’s imagine the problem had replaced with . In this new problem, you can
*imagine*solving for in terms of using the first equation, then in terms of , and then finally putting everything into the last equation to find a degree polynomial in . I say “imagine” because wow would that be ugly.But here’s the kicker: it’s a polynomial. It should have exactly complex roots, with multiplicity. That’s a lot. Really?

So here’s a hint you might take: there’s a good reason this is over but not . Often these kind of things end up being because there’s an inequality going on somewhere, so there will only be a few real solutions even though there might be tons of complex ones.

- Okay, but there’s an even more blatant thing we don’t know yet:
*what is the answer, anyways*?This was more than a little bit embarrassing. We’re half an hour in to the problem and thoroughly stuck, and we don’t even have a single that works? Maybe it’d be a good idea to fix that, like,

*right now*. In the simplest way possible: guess and check.It’s much easier than it sounds, since if you pick a value of , say, then you get from the third equation, from the first, then check whether it fits the second. If we restrict our search to integer values of , then there aren’t so many that are reasonable.

I won’t spoil what the answer is, other than saying there is an integer triple and it’s not hard to find it as I described. Once you have these two meta-considerations, you suddenly have a much better foothold, and it’s not too hard to solve the problem from here (for a USA TST problem anyways).

I pick this example because it really illustrates how hopeless repeatedly using hard techniques can be if you miss the right foothold (and also because in this problem it’s unusually tempting to just think that more manipulation is enough). It’s not *impossible* to solve the problem without first realizing what the answer is, but it is certainly way more difficult.

What this also means is that, in the after-math of a problem (when you’ve solved/given up on a problem and are reading and reflecting on the solution), you should also add soft techniques into the list of possible answers to “how might I have thought of that?”. An example of this is at the end of my earlier post On Reading Solutions, in which I describe how you can come up with solutions to two Putnam problems by thinking carefully about what should be the equality case.

Doing this is harder than it sounds, because the soft techniques are the ones that by definition won’t appear in most written solutions, and many people don’t explicitly even recognize them. But soft techniques are the things that tell you which hard techniques to use, which is why they’re so valuable to learn well.

In writing this post, I’m hoping to make the math contest world more aware that these sorts of non-formalizable ideas are things that can (and should) be acknowledged and discussed, the same way that the hard techniques are. In particular, just as there are a plethora of handouts on every hard technique in the olympiad literature, it should also be possible to design handouts aimed at practicing one or more particular soft techniques.

At MOP every year, I’m starting to see more and more classes to this effect (alongside the usual mix of classes called “inversion” or “graph theory” or “induction” or whatnot). I would love to see more! End speech.

]]>
**tl;dr** I parodied my own book, download the new version here.

People often complain to me about how olympiad geometry is just about knowing a bunch of configurations or theorems. But it recently occurred to me that when you actually get down to its core, the amount of specific knowledge that you need to do well in olympiad geometry is very little. In fact I’m going to come out and say: **I think all the theory of mainstream IMO geometry would not last even a one-semester college course**.

So to stake my claim, and celebrate April Fool’s Day, I decided to **actually do it**. What would olympiad geometry look like if it was taught at a typical college? To find out, I present to you the course notes for:

Undergrad Math 011: a firsT yeaR coursE in geometrY

It’s 36 pages long, title page, preface, and index included. So, there you go. It is also the kind of thing I would never want to read, and the exercises are awful, but what does that matter?

(I initially wanted to post this file as an April Fool’s gag, but became concerned that one would not have to be too gullible to believe these were actual course notes and then attempt to work through them.)

]]>Po-Shen Loh and I spent the last week in Bucharest with the United States team for the 11th RMM. The USA usually sends four students who have not attended a previous IMO or RMM before.

This year’s four students did breathtakingly well:

- Benjamin Qi — gold (rank 2nd)
- Luke Robitaille — silver (rank 10th)
- Carl Schildkraut — gold (rank 8th)
- Daniel Zhu — gold (rank 4th)

(Yes, there are only nine gold medals this year!)

The team score is obtained by summing the three highest scores of the four team members. The USA won the team component by a lofty margin, making it the first time we’ve won back to back. I’m very proud of the team.

RMM 2019 team after the competition (taken by Daniel Zhu’s dad):

McDonald’s trip. Apparently, the USA tradition is that whenever we win an international contest, we have to order chicken mcnuggets. Fortunately, this time we didn’t order one for every point on the team (a silly idea that was unfortunately implemented at IMO 2018).

The winner plate. Each year the winning country brings it back to get it engraved, and returns it to the competition the next year. I will have it for the next while.

And a present from one of the contestants (thanks!):

Amy and Bob play a game. First, Amy writes down a positive integer on a board. Then the players alternate turns, with Bob moving first. On Bob’s turn, he chooses a positive integer and subtracts from the number on the board. On Amy’s turn, she chooses a positive integer and raises the number on the board to the th power. Bob wins if the number on the board ever reads zero. Can Amy prevent Bob from winning?

I found this to be a cute easy problem. The official solution is quite clever, but it’s possible (as I myself did) to have a very explicit solution using e.g. the characterization of which integers are the sum of k squares (for ).

Let be an isosceles trapezoid with . Let be the midpoint of . Denote by and the circumcircles of triangles and , respectively. The tangent to at and the tangent to at intersect at point . Prove that is tangent to .

There are nice synthetic solutions to this problem, but I found it much easier to apply complex numbers with as the unit circle, taking as the phantom point the intersection of tangents. So, unsurprisingly, all our team members solved the problem quite quickly.

I suspect the American students who took RMM Day 1 at home (as part of the USA team selection process) will find the problem quite easy as well. Privately, it is a bit of a relief for me, because if a more difficult geometry had been placed here I would have worried that our team selection this year has (so far) been too geometry-heavy.

Let be a positive real number. Prove that if is sufficiently large, then any simple graph on vertices with at least edges has two (different) cycles of equal length.

A really nice problem with a short, natural problem statement. I’m not good at this kind of problem, but I enjoy it anyways. Incidentally, one of our team members won last year’s IOI, and so this type of problem is right up his alley!

Show that for every positive integer there exists a simple polygon (not necessarily convex) admitting exactly distinct triangulations.

A fun construction problem. I think it’s actually harder than it looks, but with enough time everyone eventually catches on.

Solve over the functional equation

I found this problem surprisingly pernicious. Real functional equations in which all parts of the equation are “wrapped by f” tend to be hard to deal with: one has to think about things like injectivity and the like in order to have any hope of showing that f actually takes on some desired value. And the answer is harder to find that it seems — it is (rightly) worth a point even to get the entire answer correct.

Fortunately for our team members, the rubric for the problem was generous, and it was possible to get 4-5 points without a complete solution. In the USA, olympiad grading tends to be even harsher than in most other countries (though not as Draconian as the Putnam), so this came as a surprise to the team. I jokingly told the team afterwards that they should appreciate how I hold them to a higher standard than the rest of the world.

(Consequently, the statistics for this problem are somewhat misleading — the average score makes the problem seem easier than it actually is. In truth there were not that many 6’s and 7’s given out.)

Find all pairs of integers , both greater than , such that the following holds:

For any monic polynomial of degree with integer coefficients and for any prime , there exists a set of at most integers, such that

contains a complete residue system modulo (i.e., intersects with every residue class modulo ).

Unlike problem 5, I found this problem to be quite straightforward. I think it should have been switched with problem 5. Indeed, all our team members produced complete solutions to this problem.

So I am glad that our team has learned by now to try all three problems seriously on any given day. I have seen too many times students who would spend all their time on a P2 and not solve it, only to find out that P3 was of comparable or easier difficulty.

]]>So since this is someplace between version 1 and the (hopefully eventually) version 2, it seems appropriate to call it **version 1.5**. The biggest changes include a complete rewrite of the algebraic geometry chapters, new parts on real analysis and measure theory, and a reorganization of many of the earlier chapters like group theory and topology, with more examples and problems. There’s also a new chapter 0 entitled “sales pitches” which gives an advertisement for each of the parts later. The obvious gaps: the chapters on probability are yet to be written, as is some more algebraic geometry. The updated flowchart from the beginning of the book is pictured below.

You can download the latest version from the usual page, or directly from https://usamo.files.wordpress.com/2019/02/napkin-v15-20190220.pdf. The number of errors has doubtless increased, and corrections are comments are more than welcome.

Incidentally, this seems as good a time as any to mention two more things:

- My personal website has seen some updates and re-organization, including most notably the OTIS lecture notes that I promised last Christmas, as well as my personal USAMO/IMO solutions.
- Also, I now have a
**public Facebook page**. Right now I mostly plan to use it as a mirror for this blog, but I might also find some other uses for it later. Please feel free to like me ;)

That’s all. Hope you all like it! Best wishes from the Zurich airport.

]]>*That NyQuil stuff really works*, I think to myself, and crawl out of bed. (Even though it’s like trying to drink mouthwash.) I haven’t slept that soundly all week. Or maybe I’m finally slowly recovering from my cold, and that’s why that night was better? All I know is that I’m glad I didn’t spend another night coughing my lungs out and struggling to get some shut-eye.

I drag my sorry butt out of bed and head over to my nearby computer. It’s the Friday morning before Harvard-MIT math tournament, which means that the server is getting more traffic than usual, and I was supposed to have upgraded the server in anticipation yesterday night. But Valentine’s Day was too hectic for me this year, and I never got around to it.

Not hectic for any romantic reasons. It was because I had 5.5 hours of class more or less consecutively, after which I rushed back to my place to teach for another four hours straight, all the while coughing like a banshee. Okay, so maybe I would have slept fine without the NyQuil.

Officially, this is supposed to be the software team’s job. But the HMMT website has become a complete mess that I think I might be the only person left that still knows more than half of what it’s doing. (Well, actually, Banana seems have figured out a lot of it too.) It is sort the equivalent of Frankenstein’s monster, with parts being sewn in and out over the last who-knows-how-many-years by random undergraduates with various degrees of competence, and held together by the seams with spit and prayers. The top of the main settings files still reads

Django settings for mysite project. For more information on this file, see https://docs.djangoproject.com/en/1.6/topics/settings/ For the full list of settings and their values, see https://docs.djangoproject.com/en/1.6/ref/settings/

where the “1.6” version number still makes me wince every time I see it (that means this file was created *before* I made the IMO). I am looking forward to the end of this tournament season so I can burn the whole website to the ground and re-write it.

(Making matters worse, in terms of “various degrees of competence”, I am on the low end, with no formal CS experience at all. Not good I am in charge.)

So it’s time to bump up the servers belatedly. I need to bump the web server and then the database up from t2.micro to m1.medium. There will be some downtime, but no big deal — everyone’s probably still asleep. This should only take a few minutes, and then I can work on getting the grader ready for the approximately 100,000 grading inputs that we’re going to force-feed it on Saturday.

So I push a few buttons, and let Amazon Web Services do its magic, just like the last five times I had to do this.

Something’s wrong. It’s been ten minutes already, and the website still won’t load. The upgrades should be done by now. I refresh again, and realize that it’s throwing a 500 error.

I feel a twinge of despair, which causes my cough to start to return. Looks like the NyQuil wore off. *It always starts like this. One little error, followed by another, and then…* I clench my teeth and SSH into the new server, and navigate to the log files (which I remember having to set up myself a few years ago, precisely for situations like this).

And indeed, when I get there, the same message is repeated, over and over, for the last several minutes, like a harbinger of doom:

OperationalError: (1045, "Access denied for user 'ebroot'@'xx.xx.xx.xx' (using password: YES)")

Oh, well, here we go.

My first guess is VPC groups, which have bitten me plenty in the past. Unfortunately, tinkering with these has no effect, so it seems like there is something new going on.

I double-check and triple-check the password, but it seems right. (This is the same error you get if you enter the wrong password!) So that’s not it, either.

This is a connection issue, so, the first thing I want to do is figure out whether the issue is with the upgraded server or the upgraded database. So, I go to my command bar and launch MySQL workbench, which promptly gives me

mysql-workbench: command not found

Oh, right, I don’t have Workbench installed on my desktop. So I go over to my laptop, which does have it, and try to fire up a connection to the database. This time, it gives me

喊叫 org.freedesktop.secrets 發生錯誤

Uhh, what? Okay, I have a keyring error of some sort here. I decide it’s not worth it to try to fix the workbench, and decide to do it the old fashioned way. I fire another terminal and call mysql, which promptly returns with:

mysql: command not found

Great. Okay, well, I guess I can install it, no problem. So I type in `sudo pacman -S mysql`

which gives

:: 有 2 個提供者可供 mysql：:: 軟體庫 extra1) mariadb:: 軟體庫 community2) percona-server輸入某個數字（預設=1）:

Huh? Oh, Arch Linux prefers MariaDB over MySQL, I remember now. That means SQL, unlike MariaDB, is not pre-packaged, and I’ll need to download it from the Arch User Repository and compile it form source. So I download the PKGBUILD and let it start going.

Unfortunately, this means that the binary needs to build from source, and so maybe minutes later I’m staring at the build progress and it’s at 14%. All the while the website is down, and people are starting to notice. I can feel the tension all over my body as I realize that the tournament will simply not function if I can’t get this back up and working, and the nightmare from February 2018 starts coming back to my mind again. I explode into a symphony of coughs as I struggle to gain my composure.

The compile-from-source doesn’t respond to my please. I decided that’s not good enough, and I have to do something faster.

At this point I decide I maybe should get help, so I hit up Banana who has saved HMMT on numerous occasions as well, and ask him why I can’t connect to the HMMT database. He’s done so successfully on his computer in the past, and he even has a little script he’s written to do so. His response puzzles me:

“Seems connectable. Might need to reset your DNS?”

Uh, what?

This is the only lead I have to go after, so I start prodding him like crazy, since I can’t seem to get it to do anything from my end. At first I get a couple suggestions, which don’t work, but eventually Banana finally gives me:

“Uhhhh. Can you try running db-connect on your machine and going from there? I have to start hauling.”

Right. Unlike me, Banana is actually helping the directors move around the 30 gallons of apple juice and 50 gallons of mango nectar and way-too-many gallons of water that will be used today or tomorrow. So I am really on my own.

At least I now know that it’s on the database end, sorta. I quickly locate db-connect: it’s the mini-script that Banana has been using to do the connection. Maybe this will let me get in and see what’s happening. I type:

./db-connect.sh dev

which promptly gives

mysql: command not found

Ah right, we’re still working on that, huh? The sql compilation is going nowhere fast, and so I have to think of something else.

I think of one possible approach: the workbench won’t work on my laptop, but maybe it’ll work on my desktop? I order pacman to install MySQL Workbench on my desktop too, and after a couple dozen agonizing seconds, the download is all done. To my delight, there is no error about org.freedesktop.secrets, and so I impatiently set everything up and login to find:

ERROR 1045 (28000): Access denied for user 'ebroot'@'xx.xx.xx.xx' (using password: YES)

*Oh, no, no, no.* I explode into another fit of coughs which prevent me from screaming at the monitor in frustration.

At this point, I decide this isn’t worth fighting. I can figure it out another time. For now, *the show must go on*.

That means that if I restore a backup of the old database — reverting it back to how it was yesterday, when everything went totally fine — then I can at least get the computer to work now, and worry about what the error was later.

Unfortunately, this is a painfully slow process. (The way backups work is that they don’t *replace* the existing database. Instead, it creates a brand new database somewhere on the cloud, but that has a copy of the same data as the point in time.) I load the backup, and twitch in agony as it slowly creates a new database from the image, setting everything back to how it was earlier. I hope that’s good enough.

After what feels like an eternity, the database is all set. I change the pointer of my now-working WorkBench to the new database and try to connect to see what happens, only to be greeted with 45 seconds of nothing, followed by a simple error message telling me that the connection failed.

Why? Oh yeah, I didn’t send the VPC for the new database. I do that, exhaling, it should be fine now, and connect to the new restored database, only to find:

ERROR 1045 (28000): Access denied for user 'ebroot'@'xx.xx.xx.xx' (using password: YES)

I practically choke on my own spit, which results in another several seconds of me wheezing like heck.

*Nooooooooooooooooooooooo.*

Okay, I have to fix this now.

The only clue I have is that the database script from Banana still works. But on my computer I can’t for some reason run it.

I go back to Google (which I have been using extensively the whole time), and then after another few minutes of frantic searching, realize that MariaDB is actually good enough for me: once I have that installed, I’ll have a (slightly different) SQL client, but the script should work. Hmm.

Since MariaDB is pre-packaged, that means the installation is easy, and I run it. This might be it. I fire the script again, and run into:

mysql: unknown option '--enable-cleartext-plugin'

Oh, huh. Okay, well, maybe it doesn’t matter. I delete that flag from the script, and get

mysql: unknown variable 'ssl-mode=VERIFY_IDENTITY'

Uhh. Let’s cross our fingers that doesn’t matter either? I try that again, and — much to my amazement — the connection works.

I start examining the db-connect script closely, and see that instead of the user `ebroot`

it’s connecting using some user name `dev`

. So maybe there is some new permission issues with `ebroot`

? With my new connection, I try to

GRANT ALL PRIVILEGES ON *.* TO 'ebroot'@'%';

which then returns with the following message:

ERROR 1045 (28000): Access denied for user 'dev'@'%' (using password: YES)

Uhhh.

Okay, maybe something else. I read the db-connect script again. Is there anything else that’s different? Well, there’s one more change in the code that I can at least work with: there is a CA certificate that’s being used.

I fire up WorkBench again, and try to log in again, but this time I pass a newfound CA certificate (appropriately named *rds-combined-ca-bundle.pem*) and to my relief, I find that I can now log in as ebroot. That’s the issue!

Except I have no idea how to make Django do that and I have no intention if finding out. But another Google search suggests the answer: now that I’m finally connected with ebroot I type

ALTER USER 'ebroot'@'%' REQUIRE NONE;

And breathed a sigh of relief when I refreshed hmmt.co, and harmony was restored in the world.

I looked at the clock. It was 11am. There goes my whole morning. I go downstairs to drink a bottle of Soylent for breakfast, resolving to never become a programmer for a living, or at least to get some proper training first if I ever consider it.

Meanwhile, the rest of the Harvard-MIT math tournament go on with their day, blissfully unaware of the debacle narrowly averted.

]]>