A completely routine tech support ticket has uncovered unexpected bans of IP addresses of Protonmail — a very useful service for people valuing their Internet freedoms — in several regions of Russia. I seriously didn’t want to sensationalize the headline, but the story is so strange and inexplicable I couldn’t resist.
TL;DR
Disclaimer: the situation is still developing. There might not be anything malicious, but most likely there is. I will update the post once new information comes through.
MTS and Rostelecom — two of the biggest Russian ISPs — started to block traffic to SMTP servers of the encrypted email service Protonmail according to an FSB request, with no regard for the official government registry of restricted websites. It seems like it’s been happening for a while, but no one paid special attention to it. Until now.
All involved parties have received relevant requests for information which they’re obligated to reply.
UPD: MTS has provided a scan of the FSB letter, which is the basis for restricting the access. Justification: the ongoing Universiade in Krasnoyarsk and “phone terrorism”. It’s supposed to prevent ProtonMail emails from going to emergency addresses of security services and schools.
UPD: Protonmail was surprised by “these strange Russians” and their methods for battling fraud abuse, as well as suggested a more effective way to do it — via abuse mailbox.
UPD: FSB’s justification doesn’t appear to be true: the bans broke ProtonMail’s incoming mail, rather than outgoing.
UPD: Protonmail shrugged and changed the IP addresses of their MXs taking them out of the blocking after that particular FSB letter. What will happen next is open ended question.
UPD: Apparently, such letter was not the only one and there is still a set of IP addresses of VOIP-services which are blocked without appropriate records in the official registry of restricted websites.
We love our Habr users because they’re very tech-savvy. They understand what “computer hygiene” means. Some of our users have been using Protonmail — an “encrypted email” service. We’ll discuss only the technological matter today, leaving discussion of the service itself and its business model aside.
Every day we send out a lot of emails to our users, and since we care about our independence and their privacy, we don’t use external mail services (ESPs). Instead we use our own resources, from bare-metal servers and self-maintained MX servers to encrypting connections and owning our independent IP addresses.
Last week our support team was overloaded by messages from Protonmail users, complaining that our mail doesn’t get through to them:
Hello. Since around the first week of March 2019, when I try to login, I get a red banner saying that they couldn’t send an email to my address. I tried to send a test message manually, to no avail.The mailbox itself, hosted by Protomail, is perfectly functional (other emails do come through fine). The last digest from Habr is from February 28th.
I didn’t change any of the settings neither there nor on Protomail, but around the time the trouble first came up I was logged out of the Android app.
I don’t think the account has been compromised, but I couldn’t find the list of IPs used to access it so I can’t be sure. I hope for your help, since without a working email I can’t vote on comments/posts.
Changed the email address from Gmail to Protomail. The confirmation email doesn’t arrive to the new address.
Of course, our tech support suggested the simple stuff like checking spam folders, but the sheer volume of similar complaints forced us to dig deeper.
For most modern Internet users using email means logging into the “personal Inbox” on the email service provider’s website and then sending letters through the same web interface. Then some magic happens and a couple of moments later the letter arrives to the web interface on the receiving end.
This “magic” is called SMTP (or esmtp, to be precise). The sending server extracts the domain part (after the @) from the receiving address and makes a DNS request for MX servers of the receiver’s domain. For support@habr.team, it looks like this:
MX stands for “mail exchange”. It indicates the email service used by the receiver or, to be precise, what email servers the receiving domain’s hosting collects new mail. The example above shows that our domain, habr.team, is hosted by Google (G.Suite).
After finding the MX servers, a request is made through the esmtp protocol to the server with the highest priority, to deliver the mail to the user. Multiple servers are listed for redundancy, since “interconnectivity” of the Internet is a very relative term.
That’s how sending a letter looks like:
NB: mail from a certain domain doesn’t necessarily have to be sent to users on MX servers listed in the DNS; this is only used for incoming mail. Outgoing mail can be forwarded through other servers, usually listed in an SPF record.
We dug through our mail logs and discovered that connection attempts from our servers to Protomail’s MX servers (185.70.40.101, 185.70.40.102) always timed out. This looked strange for a number of reasons and resembled the mechanism for Internet censorship in Russia.
In general, the term “Internet” is made of two words: “Inter-Net”, and can be interpreted as “network of networks” or “united networks”. The Internet doesn’t have a “technical center” (though it does have an “organizing center”): it simply connects different networks that are, in theory, equal to each other (though some networks are more equal than others, but that’s a story for another day). Networks are called “Autonomous systems” (AS) and are connected to each other via gates, or “peers”. Each AS has a unique number used to identify it by other ASes. Like IP addresses, but in a more general way. Each network receives from its “neighbours” the topology of its connections to nearby networks, how these nearby networks connect to their nearby networks, etc. Essentially, each network has a map of how all AS connect with each other from the perspective of that network. A route from one AS to another according to that map is simply called “AS path”.
For example, our autonomous system number (ASN) is 204671, Protonmail servers are hosted on the network of a large American corporation Neustar, and its ASN is 19905. We have two gates with ISPs, meaning two possible AS paths from us to the Neustar network. For a number of reasons, one of the gates (through MGTS) is preferable, so our AS path looks like this: 204671 (us) — 57681 (MGTS, the ISP), 8359 (MTS, the larger ISP) — 22822 (Limelight) — 19905 (Neustar).
And here’s the routing table:
Each traceroute to any of Protonmail’s two MX servers cut off at the MTS network and looked like this:
GW-Core-R3#traceroute ip 185.70.40.101 probe 1 timeout 3
Type escape sequence to abort.
Tracing the route to 185.70.40.101
VRF info: (vrf in name/id, vrf out name/id)
1 185.2.126.73 [AS 57681] 2 msec
2 212.188.12.73 [AS 8359] 2 msec
3 195.34.50.73 [AS 8359] 3 msec
4 212.188.55.2 [AS 8359] 3 msec
5 *
6 *
7 *
8 * We found an alternative host within the Neustar network and used it as reference to eliminate possible disruptions between MTS and Limelight:
GW-Core-R3#traceroute ip 156.154.208.234 probe 1 timeout 3
Type escape sequence to abort.
Tracing the route to 156.154.208.234
VRF info: (vrf in name/id, vrf out name/id)
1 185.2.126.73 [AS 57681] 2 msec
2 212.188.12.73 [AS 8359] 2 msec
3 212.188.2.37 [AS 8359] 14 msec
4 212.188.54.2 [AS 8359] 20 msec
5 195.34.50.146 [AS 8359] 27 msec
6 195.34.38.54 [AS 8359] 37 msec
7 68.142.82.159 [AS 22822] 26 msec
8 *
9 156.154.208.234 [AS 19905] 26 msecMeanwhile, we successfully completed another trace through another ISP to Protonmail’s MX servers (it cuts off at Neustar, but that’s expected — the connection still works):
$ traceroute -a 185.70.40.101
traceroute to 185.70.40.101 (185.70.40.101), 64 hops max, 52 byte packets
1 [AS49063] hidden (hidden) 5.149 ms 268.571 ms 6.707 ms
2 [AS49063] 185.99.11.146 (185.99.11.146) 5.161 ms 6.317 ms 5.476 ms
3 [AS0] 10.200.16.128 (10.200.16.128) 5.588 ms
[AS0] 10.200.16.176 (10.200.16.176) 5.225 ms
[AS0] 10.200.16.130 (10.200.16.130) 5.001 ms
4 [AS0] 10.200.16.49 (10.200.16.49) 6.480 ms
[AS0] 10.200.16.156 (10.200.16.156) 5.439 ms 7.469 ms
5 [AS20764] 80-64-98-234.rascom.as20764.net (80.64.98.234) 6.208 ms 9.301 ms 6.348 ms
6 [AS20764] 80-64-100-102.rascom.as20764.net (80.64.100.102) 24.281 ms
[AS20764] 80-64-100-86.rascom.as20764.net (80.64.100.86) 54.632 ms 23.936 ms
7 [AS20764] 81-27-254-223.rascom.as20764.net (81.27.254.223) 27.589 ms 116.438 ms 27.348 ms
8 [AS22822] siteprotect.security.neustar (68.142.82.153) 28.683 ms 25.376 ms 41.489 msGiven that traceroute isn’t a very reliable tool, we conducted another couple experiments, for example, with MTS’s Looking Glass service:
It became clear that it’s probably an intentional restriction of service at the MTS level. However, consulting with Roskomnadzor’s official registry revealed that both addresses (neither domain name no IP) are not listed there:
Unable to find anything on your request
Leaving specifics of Internet censorship in Russia aside, there’s only one valid justification for an ISP to block a resource — the so-called “registry dump” containing the resource put there more or less legally. Sometimes resources get blocked without a relevant registry entry (they’re colloquially named “registry-less blocking”), and usually the justification for these doesn’t stand a chance in any legal case.
At this point, we suspected a simple technical misunderstanding or a botched unlock of another unrelated site. Yes, we don’t sound the alarms without meticulously fact-checking everything first.
We sent an email detailing our findings to MGTS tech support and asked for clarification. A bit later we received a non-answer: “it’s not us, it’s MTS, ask them”. Of course, we didn’t, but instead forced MGTS to do their job and investigate it properly. The response we got was very interesting: according to an MTS employee from the relevant department, they’ve been contacted by the FSB via an official letter №12/T/3/1-94 from 25 Feb 2019, demanding them to urgently block these hosts:
At this point, our bullshit detectors have gone off the scale and we dug even deeper. And faster.
We sent a request to the FSB asking if the letter exists and if it does, what justification do they have:
We asked MGTS to provide a justification as well:
After that, we went to some topical chat rooms in a certain illegal in Russia instant messaging service. The telecom community reacted pretty reluctantly:
- “I had experience solving these issues and no one wants to use RKN’s tools. First, it’s complicated. Second, transferring the problem to another department doesn’t solve the problem”.
- “You need to provide so much documentation and jump through so many bureaucratic hoops (and there’s monetary punishment for not doing all of that) that no one bothers”.
Well, it’s hard to really judge them, seeing as those working in telecoms have to deal with so much crap (considering “SORM”, “network node” or “registry dump” aren’t just words to them, but a daily annoyance).
Nevertheless, the Russian Internet Defence Community’s chat room took the case enthusiastically and conducted a proper investigation of their own.
They suggested an interesting idea — to check what ISPs (in Russia and otherwise) block access to the MX servers through RIPE Atlas. The results were predictable, but still very curious: in Russia, the servers are blocked by MTS and Rostelecom, as well as networks working through these two ISPs (results on the primary MX server, and the backup one). The worldwide check detected issues in Russia, Ukraine and Iran (worldwide results for the primary server, and the backup).
A more involved research showed that Rostelecom acts in a similar manner:
After the weekend, MTS finally provided a scan of the FSB letter that ordered the blocks. Of course, they blamed “telephone terrorists” and tied it all up to the XXIX World University Winter Games — Universiade 2019:
Then Protomail representatives reacted on reddit, Twitter and TechCrunch. As expected, they were surprised by the bluntness of FSB’s methods and offered to cooperate toward finding real criminals:
Hmm. Protomail themselves suspected there’s a hidden agenda to all of this. Well, it appears to be the case. So, as I’ve already explained avobe, MX records are a mechanism for handling incoming emails. FSB clearly deliberately broke the incoming mail rather than outgoing, so their, ahem, “justification” of saving school headmasters from trouble is very clearly fabricated. So, we have three options:
- They simply blocked what they first found (a lazy explanation, but the most probably one according to the Occam Razor: someone must have just read “nslookup for dummies”);
- They tried to limit the possibility of setting up anonymous and untrackable Proton addresses collecting damaging information on themselves (doesn’t work in the vast majority of cases)
- The cursory UFO explanation.
Here’s the proof: an email sent from Proton to another service went through different IPs that aren’t blocked. Remember, FSB banned 185.70.40.101 and 185.70.40.102. Do you see these here?
Head of ProtonMail confirmed the findings to TechCrunch and suggested to fight terrorist activity in cooperation with foreign law enforcement, rather than just handwaving the issue away:
ProtonMail chief executive Andy Yen called the block “particularly sneaky,” in an email to TechCrunch.
“ProtonMail is not blocked in the normal way, it’s actually a bit more subtle,” said Yen. “They are blocking access to ProtonMail mail servers. So Mail.ru — and most other Russian mail servers — for example, is no longer able to deliver email to ProtonMail, but a Russian user has no problem getting to their inbox,” he said.
“The wholesale blocking of ProtonMail in a way that hurts all Russian citizens who want greater online security seems like a poor approach,” said Yen. He said his service offers superior security and encryption to other mail providing rivals in the country.
“We have also implemented technical measures to ensure continued service for our users in Russia and we have been making good progress in this regard,” he explained. “If there is indeed a legitimate legal complaint, we encourage the Russian government to reconsider their position and solve problems by following established international law and legal procedures.”
— ProtonMail chief executive Andy Yen @ TechCrunch
Also, it looks like this FSB letter only ordered to block SMTP servers for incoming mail. Web access and outgoing SMTP servers still work, which means that whatever the FSB tried to do, they weren’t very good at it.
We’ll say again: even disregarding the entire legal side of the issue of regulating the Internet on the very particular 1/8th of Earth land, there are rules of the game. The rules aren’t terribly clear, very ambiguous and they change all the time, but they’re still rules, even if they’re clearly designed to benefit the maintainers of these rules. And even then, there are people trying to bypass them. This was very Kafka-esque, with no due process at all. At least in any court case you can call upon an industry expert for consultation, but there the decision was purely based on the personal worldview of one particular person.
So, here are all the facts we’ve gathered up to this point. All the requests have been sent, but not all have been responded to. Of course, we hoped that it was just a consequence of a botched job by someone at MTS, but, to be honest, it didn’t look terribly probable from the beginning.
As for our users who also use Protomail, then they can safely continue to use their Proton mailboxes with Habr, since we rerouted the traffic from us to Protomail service through another Russian ISP that doesn’t play these sorts of games. And MGTS is probably about to lose another customer.
