At 12:15 a.m. yesterday my phone lit up; emails and text messages poured in one after another — network traffic alerts, CPU alerts. There must be something unspeakable behind this. Why do hundreds of threads keep dropping? Why do callbacks take forever to return? Why does MySQL keep getting assaulted? Who’s behind this case of nested loops within nested loops? Behind the bizarre bugs — man or ghost? What’s hidden in objects arriving empty every single time? Type conversion errors, or null pointer exceptions?
Let’s step into the twisted inner world of programmers. Welcome to Ren Fei’s blog.
As of publication my program is the second generation, renfeid 2.0, which already has fairly complete tracing and auditing — though no rate limiting yet. Rate limiting comes in the third generation, renfeid 2022, working together with the WAF.
Reconstructing the Timeline
Leaning on my development experience, the system keeps fairly thorough audit records of every user action, producing this timeline:
- 2022-10-18 23:36:52 Arrived at the word-segmentation tool page through Google search
- 2022-10-18 23:49:00 Tested the segmentation API 4 times using Postman 7.29.2
- 2022-10-18 23:52:04 Started calling the segmentation API in bulk via okhttp 4.10.0
- 2022-10-19 00:15:31 Server network traffic anomaly alert
- 2022-10-19 00:33:21 Server CPU usage alert
- 2022-10-19 00:40:04 Bulk calls stopped
- 2022-10-19 00:48:16 Visited the segmentation tool page once in a browser
The visitor’s IPs were 60.12.8.240 and 60.12.8.241, two China Unicom public addresses located at Jucai Road, Binjiang District, Hangzhou, Zhejiang.
The whole episode lasted 48 minutes: 290,653 total requests, 311.8 MB of traffic consumed. The peak fell in the second of 2022-10-19 00:14:16 with 434 requests; overall average TPS: 208.
Motive
Writing a program specifically to hammer my API — what’s the motive? Let me psychoanalyze the programmer’s twisted inner world.
- Guess 1: Laziness, a programmer’s virtue. He had a batch of content to segment, searched for a segmentation tool, landed on my site, grabbed the API, and called it directly.
- Guess 2: Provocation. Programmers always look down on other people’s work. You’re that good? Let me test whether you hold up. Write an infinite loop and see how strong you really are.
- Guess 3: Grudge. Someone was refused somewhere and started retaliating — hunting down the most resource-hungry endpoint and hammering it repeatedly, a CC-style attack.
- Guess 4: Deliberate reconnaissance. Probing the site’s capacity to plan a larger attack, then hitting it continuously and negotiating with the webmaster — free ad placement or a link exchange.
- Guess 5: Wandering around. He happened to have a similar development need, got curious, and took a run at it, using the performance numbers as his own reference.
I lean toward guess 1. I looked at the content being segmented: it seems to be running a dictionary through it, since the data was ordered and sorted by pinyin. Reconnaissance for an attack is plausible too — in which case I can only wait until the third-generation renfeid 2022 goes live, with the WAF banning IPs in real time.
Damage Assessment
Having been attacked several times before, I’d already switched from a paid CDN to a free one, so those ~300,000 requests cost me nothing financially — 311.8 MB of traffic works out to roughly 0.24 yuan. It happened in the middle of the night when almost no other users were on the site, so I didn’t intervene.
How to Handle It
Against API hammering, the most effective tool is rate limiting. Count how many times an IP calls the API per minute, hour, and day; reject it once it crosses a threshold, and once it crosses a ban threshold, block it at the firewall.