This is a question on Quora, where Mick’s answer got 13.5k “upvotes”. He recalls:
A psychology PhD once hired me to track down a bug. The program was written by one of his students, and it would often produce strange output. The program’s function was to read data from a file, ask 50 questions, do a series of calculations, and then produce a score based on the PhD’s research. The program ran on a 3B2 at the university. He demonstrated the program to me and confirmed the bug was reproducible — every time it switched between questions, some strange characters would flash by. I thought this should be simple, so I agreed, and we agreed to be paid by the hour.
Day 1
I came to this 3B2, logged in with the student’s account, found the C source code, and started testing. The code was very poorly readable — all the code was written on one line, and the variable names were three random letters! I’m glad I chose hourly billing. I formatted the code into my habitual style so it was at least somewhat readable.
After that, I used the curses library to move to a point on the screen, print a question and answer, then wait for a response. But after printing the first line, some garbled characters appeared, and after about 1/2 second the garbled characters were overwritten by the question. This should be easy to fix — there were only five places that print information, and the garbled characters appeared in all of them. No big deal, just delete mvpwintw() and it should be ok. I deleted it and started compiling, thinking the problem was basically solved. But when I ran it, the interference came back! Only the garbled characters changed, but the symptom was the same!
I checked the code and found it had reverted to how it was before my changes! 15 files, chaotic formatting, three-letter variables. Why didn’t I back up the code back then — I could shoot myself. I formatted them again, this time putting the code into three files with different names. Then I backed up the whole folder and set the permissions to read-only. After compiling, everything was fine. As soon as I ran it, 15 files appeared in the folder again! My modified source code wasn’t deleted, but the interference came back.
I realized the code must be somewhere on the disk, and when compiling it adds the program to my modified code. So I prepared to search the include area (/usr/include), because we were using a research version, so all code except the kernel was on the machine. There were too many header files, so searching on the 3B2 took some time. That was the work of day one.
Day 2
The disk search yielded nothing, which meant the garbled characters were either encrypted or somewhere in the lib. But I didn’t find them. I decided to search all text files; this time it took longer than yesterday. Day two ended like that.
Day 3
No result. The strings were encrypted. I had to check bit by bit according to all the header files. This would take quite a while; at the same time we warned the school that someone might have obtained root on Dr. Phelps’s computer. But they didn’t care — probably just a lab computer.
I opened the #include files but found no code. Later I found they were all compiled into one file. No big deal — we had the source code anyway, at worst we’d recompile all the libraries.
Days 4-6
Next came the hardest part. We finally explained the problem to the school’s nerds. Then we got Mark (I think he became a Unix admin purely because he married the Dean’s daughter) to start learning to compile. Finally he agreed to let me do it, because he couldn’t do anything. At the end of day 6, the compilation was finally done.
I took out the modified code and started recompiling. Everything was fine, then I ran it — damn! The problem came back. The source code was split into 15 files, and the interference came back. It was like magic; I felt defeated. The problem definitely wasn’t in the source code. Dr. Phelps was also a bit unhappy; he felt that with so much time, even rewriting it should have been done. “Of course,” I said dejectedly, “you’re right, maybe rewriting is better.” “OK, we’ll start rewriting tomorrow,” said the doctor.
Day 7
Go to hell, I won’t give up! I told Dr. Phelps: “You don’t have to pay me; just give me time, and I must find this bug.”
Days 8-14
I got smart — he must have modified some library. I started studying the compiled assembly (though I didn’t understand assembly before), from starting to learn to finally understanding the assembly code, which took six days. Although I found nothing abnormal, it was a total waste of time.
Day 15
Suddenly, I realized the problem might be in the compiler — it must be. Every time the code was compiled, the compiler would add interference to the source code. I’d heard of this kind of situation before.
Aha! I found it! We also had the compiler’s source code; I checked it, and thank God, finally found it. The code in the compiler linker was like this:
- Detect all calls to fopen(), search the opened file for Dr. Phelps’s questions; if found, 2) when compiling, rewrite 15 files; 3) use these 15 files to compile the doctor’s program, and output a name in -o form at link time.
The compiler had been modified by this student to add code to Dr. Phelps’s program.
A few days later, AT&T’s technical support provided the original compiler and linker code; we recompiled and replaced the modified compiler and linker.
But the problem wasn’t solved. The compiler was contaminated by other source code we didn’t have. This code existed in the now-executable compiler; when compiling the compiler, it would add the contaminating code. But it didn’t modify the code in /usr/src; instead it copied it to a hidden folder, modified the compiler source code, compiled, and finally deleted the hidden folder. It took AT&T a long time to discover this. The student had modified the compiler so that when recompiling it would add the contaminating code. Finally we had to copy the compiler’s bytecode file version from another 3B2 machine to finally solve the problem.
Through the compiler’s code we also found that if /sbin/login was compiled, some backdoor code would be added, allowing anyone to log into root with a specific password. This computer could be accessed via modem or Tymnet. Finally, this attracted the school’s attention.
This person is truly a genius, but also truly terrifying!