Democratizing Internet Monitoring with IODA

Understanding what's happening on the Internet from a macro level gives us incredibly useful insight into much of what's happening in the world at any given time. And that's from major power disruptions, governments shutting down communication, natural disasters, and even insight into group behavior, like who's using what applications in what countries and when.

Well, there's a project out of Georgia Tech called Internet Outage Detection and Analysis or IODA that ingests a variety of information to analyze activity on the Internet on a global scale. And then they provide that information to pretty much whoever needs it, and that's including governments, private industry, advocacy groups, via their online dashboard, and therefore, anyone who can access their website.

So joining us today are Amanda Meng and Zach Bischof, both at Georgia Tech and working on the IODA project to talk about how it started, how it actually works from a technical perspective, and what some of the goals are and the vision is for the future. And, of course, also joining is Doug Madory, a friend of Amanda and Zach, and the IODA project. So this is definitely gonna be a good one. My name is Philip Gervasi, and this is Telemetry Now.

Amanda, Doug, and Zach, thanks so much for joining today's episode. And, Doug, thanks so much for making the connection, for for all of us here, and and it's really a pleasure to meet you, Amanda and Zach.

So as we kick off today, I'd really like to get an understanding of, what the IODA project is all about. But I'd like to I'd like to understand your specific your personal role at, at at Georgia Tech, how you got to be in those roles, and, and what you're working on right now. So why don't we start off with that? If you can give me a little bit about your background. And, you know, Amanda, why don't we start with you?

Great. Thank you so much for having me.

My name is Amanda Ming and I'm a research scientist at Georgia Tech. I am a social scientist, so I've but I've always worked with computer scientists and I am interested at the intersect of, like, civics and technology or civic data. Very interested in, data literacy and civic literacy and how those things kind of interact with each other. And I've studied data activism and made my way to the IODA project because we have a lot of advocacy organizations and digital human rights researchers that make use of our data to monitor for shutdowns and advocate for Internet freedom.

So that's certainly something that draws from my from my background. So that's a little bit about me and and and how I joined the team.

Great. Thanks. And, yeah, I mean, there's a lot of overlap with, what what Doug, and I discuss quite often actually, both on this podcast, but also just in our professional lives and how his analysis of both from the technical side, and also what's going on in, like, the geopolitical, climate of the world today. So really interesting.

Zach, how about you?

Yeah. So my background might be a little bit longer, but, my my background is in Internet measurement and computer science.

Initially, when I started grad school over fifteen years ago, I was interested in, you know, better understanding the Internet and I I had a particular interest just with some of, like peer to peer networks and BitTorrent.

At the time, early in grad school, I was helping work on this BitTorrent plug in that was trying to improve performance and, also detect, like, network events in a kind of a coordinated, collaborated way.

And so we started collecting, different data from all these different, trackers and BitTorrent sources.

And so how I actually kind of got into this general area of research was, and there'll be some parallels with IODA here, is using this BitTorrent data to actually identify some of the disruptions in Internet connectivity during the Arab Spring back in twenty eleven.

And then I also was, you know, using this activity to, kind of analyze the impact of the twenty eleven Tohoku earthquake and tsunami in Japan, and saw the you know, those regions that were most heavily impacted, by the tsunami were the ones that saw, like, the largest drops in BitTorrent behavior or activity, I guess.

And so since then, I I've just always kind of been in interested in this, end user perspective and understanding kind of the health of the Internet.

Interestingly, at the same time, Alberto Dinotti, who's our PI in the lab who leads the IODA project, he was at, CADAA at UC San Diego, at the time. And he also kind of was in parallel getting interested in this topic, and kind of, in parallel was using, BGP data, for example, I think at the time for seeing the disruptions in the Internet connectivity, during the Arab Spring.

So it was kind of interesting that we were working a bit in parallel on a similar topic, and then I kind of kept bumping into him at different, research conferences over the years.

I did a short stint where I was working at, IIJ research lab in Tokyo. And then when I moved back to the US, I was looking for job opportunities, and Alberto was looking for a research scientist on IODA, and, just kinda seemed like a really good fit for, the both of us to work together.

Great. And Amanda and Zach, you're both at Georgia Tech now. Right?

Right. I do work remotely from Portland, Oregon.

So Oh, really?

Okay. Remote work. Yeah. That is that is, how, I think, one hundred percent of Kentik operates as well.

So I would like to get into the project, Internet Outage Detection and Analysis project or IODA, and and start off with really just getting an understanding of what this project is all about. Not necessarily from a technical perspective, although we're certainly gonna get there. And, and I am fascinated by that, and I'd like to get into it. But what is the, what is this project all about and and your goals, of course?

I guess, I feel like, I'll let Amanda answer this one. I feel like she's got a bit more, polished answer.

Sure. So we provide a publicly accessible dashboard with Internet measurements, that give users, signals on Internet connectivity data so they can see connectivity of Internet infrastructure in their country or in their kind of subnational region or a specific network that they're interested in. And these are near real time measurements, and they could also look at historic views.

We also have a, a detection system that detects outages. So if you wanted to just go and look at the dashboard and see where in the world IODA has detected outages, you board and see where in the world IODA has detected outages, you can view that map at the country, region, or network level. And, I we have various signals that we use to measure connectivity of Internet infrastructure, and they all are slightly different techniques for measuring connectivity, which is important.

And, I'm sure Zach could kinda go into those those three. And we're also constantly working on integrating new signals into IODA, so that we can provide, you know, as complete a picture as we can of connectivity of Internet infrastructure.

Okay. How do how do folks access this publicly available dashboard?

Sure. So they can use IODA dot live. That's our kind of persistent URL. It'll redirect to our Georgia Tech, URL, but, IODA dot live.

Okay.

Now, and and maybe maybe, Zach, you would agree that we really don't need any reason to spend Friday night looking at, like, a packet capture. I mean, that sounds like an awesome evening, to me. My wife might disagree. But, I'm assuming that you're not doing this just for fun. So what is the goal of the project? Why are you doing this sort of global internet analysis?

Yeah. I I think, I mean, there's so many ways that, these kind of large scale Internet outages can affect people.

And I I mean, that is what we are trying to, you know, what the focus of the project is. And there's been lots of research efforts, for example, on, you know, monitoring individual, like, slash twenty four IPv four blocks of the Internet, trying to observe activity levels and things like that.

And I I think, you know, individually, like, you can monitor the all of these, but really kind of taking all these different data sources and kind of, coalescing them into one comprehensive view of Internet connectivity, I think is, really one of the big goals, to, you know, focus on those macroscopic events.

You know, of course, if we focus on individual block outages or, you know, even sometimes specific cities, there can be so many things going on that I think there's so much noise and it's really hard to, you know, validate. For example, like, in in on the block of my sorry. On my street block, if there's an Internet connectivity issue with a specific provider, you know, those probably happen all the time all around the world and validating those is something that's really difficult.

But when you get to these macroscopic events, you start to see, disruptions sometimes across different signals.

And I would say that's another important thing of IODA is the fact that we do take data from a variety of different sources and combining them on one platform, because we'll see cases like, you know, a couple months ago in Chile there was a large scale power outage.

We actually saw that BGP during this was not like IODA's BGP signal was not heavily impacted but the other ones were and our guess is that you know the network infrastructure, the facilities, maybe they had power generators that were able to keep the routers online and keep them connected, but then end users were completely cut off from the Internet. Right.

And so I I would say that, you know, this goal is kind of multifaceted in that we want these large scale major events. We want to be able to understand the scale of their impact, who's impacted for how long, and then also this kind of need for, you know, a multi pronged approach to capturing these events that, you know, sometimes we'll see a disruption in one signal, but it doesn't agree with the other ones. And, you know, there could be other causes of this that first, you know, people turned off their computers to go watch the World Cup or something like that. I think we've even seen examples of that.

So, yeah, high level goal like I was saying just, really the importance of combining all these different sources and trying to focus on those major events.

Yeah. And there's certainly the the whole geopolitical nature that, Amanda, you already alluded to, which Doug I mean, you write about and speak about, a lot.

So, Doug, and you were about to say something, so forgive me for getting Yeah.

So I am a heavy user of IODA, big fan, and somebody I could speak to from a user's perspective of the value of this tool. And just like Zach was saying a moment ago, how, sometimes an outage will only manifest in one of the signals. Another example was in Spain last let's see, last week, it was just or in the last couple weeks there was an outage in Portugal and Spain. The Iberian Peninsula went down, and that showed up in active measurements. So pinging things showed that there was an outage, but the routes there was a small amount of routes that were lost. Most of the routes, the vast majority stayed up, and so that wouldn't be a very good signal, to, to use to, if you had that only, only well, if you only had BGP to work with, you wouldn't be able to grasp the the scale of the outage.

And I've been dealing with that long before IODA came along. You know, I was at other companies where we had tools doing Internet measurement and also kind of realizing the limitations of each type of dataset. We have we had actor measurement. We had BGP.

We had, we would when I was with before and after, I guess, with with Dyn DNS, we had Dyn query volume was our standard for traffic from any part of the world. And so there'd be times where the routes would be would be up. The, things that we're pinging are still answering, and then but there's an outage. And And so then we would see that see it in that we've lost all the queries coming out of a geography.

And these days, with Kentik, I've got NetFlow, so we have a good idea of traffic that we can use to try to kind of confirm or corroborate an outage. But sometimes it's a little more of an art than a science, and so you need a bunch of different tools. It's helpful to have IODA as a reference. Even though I've got a lot of tools, I still will use this because it's very easy.

And all these things, you know, I think any of these data types, you put it on a time series, they become kinda intuitive. There's some sort of a clearly, your eye sees a steady state, and then there's, you know, a cliff that it drops off. Okay. That's a good visual cue.

Something happened. It's very easy to, and so you may have you may we we may have had some of this data before IODA existed, but you'd have to pull it out and process it and make a graph. And then you'd see the thing like, well, that's all done for you. So it just, saves a whole lot of time.

And because it's so frictionless, then, you know, people, like I know I do, I know there's a lot of people members of the digital rights community who make a lot of use out of IODA.

It makes their work that much easier, and they can cover a lot more ground that way.

Yeah. Can we speak to that a little bit? Amanda, you know, you you talked about you how you're looking at how this world of Internet connectivity, intersects with, like, the geopolitical nature of how folks utilize the Internet today. And when I say folks, I mean even at the, you know, government level and and, some of the activities that Doug has covered in his analysis over the past few years have been really fascinating. So what, what is your focus, you know, using this tool and this data, in your, you know, how you, you know, analyze what's going on in the world today?

Sure. Yeah. I mean, part of what I do is actually try and train and build capacity for other organizations to make use of IODA's data. And so that is a big goal of ours, is to make it accessible, and actionable so that folks who want to monitor connectivity in their region or in their country, have the skills to do it themselves. But we also do collaborate on reports where we'll do kind of a retrospective of a shutdown or even, a cable cut or a power outage.

And it's also something that we we like to do with Doug, with the folks at Cloud for the Radar so that we're showing all the perspectives that are telling us the same thing, that this is the outage that that happened, and this is how it's showing up in all of our measurements.

So, yeah, we want to help provide information that, you know, brings clarity to folks because we've learned that when a shutdown or an outage happens there there can be a lot of uncertainty, a lot of chaos, and just having information of, measurement information, and if we can try to investigate the cause, you know, we we definitely do that. But, I think a a big goal of ours is to build capacity of these organizations to use, IODA themselves and also do as much as we can from a user experience perspective to make sure, we're we're creating a usable platform.

Yeah. And I did see some of the higher profile, organizations listed on the website.

And so certainly you're working with and serving, you know, organizations like, I think I saw the FCC, the Internet Society, Comcast, I believe, is a partner. I don't know what the relationship is there. Maybe you can help, you know, expand upon that as well. But, all of these folks are utilizing the service. Right? So you're providing the data so so that they can make informed decisions?

Yeah. For example, we we published a report last month, with Amnesty International and looking at, the shutdowns last October in Mozambique where they had worked with their in country partners to understand the protests, what had happened with, you know, police violence and some kidnapping. And we were providing the piece about Internet connectivity, and the government actually did not, you know, announce ahead of time or even quickly after the fact that they had shut down the Internet. You know, a few months later, there was a statement that there had been some interference. And so we work with organizations like that to fill in that piece of the Internet connectivity from our perspective.

But, yes, definitely, we've had the partners you've mentioned support us because they wanna see our data available, provide funding.

I think, yeah, generally, the list of sponsors that we have there or supporters of the project, you know, kind of ranges and some of them are, you know, larger grants or proposals especially from the government.

But, yeah we we've had some commercial partners in the past and, you know, at the moment we're also working with a couple of organizations. We're trying to, get some more comprehensive measurements and data sources and things like that. I guess we'll talk a bit about the future later probably. Yeah.

Yeah. Doug, maybe you remember back in the days when you were running networks. Right?

The A long time ago.

A long time ago. Yeah. No. No. Totally. Totally. But, you know, the old saying was that, the PCAP doesn't lie.

Now I know that you're not using packet captures necessarily. You have other other, telemetry. I get it. But the spirit of what I'm trying to say is that we have the data.

And so working with some of these, government entities and, you know, literally other countries around the world that are not necessarily telling all the truth there, Amanda, very interesting in that you are, that source of truth for, in this case, government shutdowns, but also, you know, other other activities that are going on, whether they be natural disasters or, you know, major power outages that we saw in as we saw in in Europe recently. So, certainly, there's a service there that's applicable to a lot of different scenarios.

But I I do wanna talk about the, technical mechanisms.

You know, you mentioned BGP. You mentioned terms.

Doug, you mentioned I think you mentioned, active, telemetry at some point.

Maybe maybe it'd be great to go through. Yep. So yeah, so this I think this one's for Zach. So, we've got we've got BGP, active measurement ping.

Sometimes I see, like, Google, like, search traffic, as a source, and then there's the background radiation. So, I think maybe those four are the main components. Okay. Yeah. Alright. Why don't we start at the top then?

It's interrogation here. So the the b g b what's the sources? Route views, right, both?

Yeah. Both. So it's a a mix of route views and ripers, collectors.

So for this, off the top of my head, I don't remember all the details but we basically there's some back end analysis to identify the what we call the full feed peers which is, that they have, kind of a comprehensive view of all the prefixes that are being advertised on BGP. I'll just try to summarize it that way. You know, there's some peers that might, have a lot of route aggregation going on that they're missing a lot of the different prefixes that get, announced on the internet.

So we identified route or something.

Yeah. Yeah. So we identify these, full feed peers, and then those are the ones that we're collecting data from from the two projects.

And we're basically just the BGP signal is, you know if you if you just count the number of prefixes advertised on the internet, it starts to get complicated because you know there's someone might have a slash eight but then you need to, you know, part of that might be a more specific prefix that's announced and so what we actually do is we try to normalize this by the size of the prefix, and so we just count the number of slash twenty four blocks and then the slash slash twenty four blocks get mapped to well, you know, obviously in AS, that's the easy one, mostly.

And then the more, kind of nuanced one is using IP geolocation to map those slash twenty four blocks to a country and subnational region.

So Let me ask let me ask a question.

Let me interject you. So, we often people who do this, and you and I are those people, you you kinda have to pick an arbitrary threshold of what is globally routed to be counted. And it and whatever whatever threshold you choose, there's gonna be some something gets got left out. How do you guys arrive at that? What's so you have, let's say, if you have Ripe and route views, then we're talking about, like, I don't know, seven hundred and v four sources or so.

So does it need to be seen by three fifty to be counted?

Or how does that work?

Yeah. So this one, we're gonna have for this one, we'd have to bring in our research developer.

But, Is it RTFM, Doug?

Yeah.

I no. I I do I, yeah. I I get the question and I always forget the specifics on this but it's, you know, so I mentioned the full feed peers.

I think they're I forget what the threshold is that it's like for something to be It's okay.

It kinda doesn't matter. I I I I've I've just I'm just curious because I like, everybody's everybody's dealt with this, and you always have to choose something. I guess what's in your favor, at least, you know, more so in v four than v six, but there's a there's almost like a drop off, like, of what's gonna be routed and then the other it's almost binary, or or multimodal where, if you were to pick I know we made a graph at one point back at Renesas Days. This is, like, fifteen years ago of just, like, are we choosing the right threshold? And we do a did a graph, and it was like, if you picked anything between, like, ten percent and ninety percent, you would arrive at basically the same thing.

So we're like, alright, I'm not gonna sweat this that much.

But it's just one of those things everybody has to choose.

I did pull up the, IODA resources page because I always forget off the top of my head. But, yeah, our definition for the full feed peer, this it's a two step thing here so I'll start there. The full feed peer is it sees more than four hundred k v four prefixes, and more than ten k v six prefixes even though we don't, right now we're not really including v six in the data.

But then for a prefix to be consistent with the whole Zack.

You're gonna get letters.

I know. I know. I'm gonna upset people. It's a future work. Future work.

Send your complaints to at zach, IODA info.

Contact dot e u.

And then our our prefix is considered, visible if more than fifty percent of the full feed peers observe that. Makes sense. Makes sense. Two step definition of, you know, what we consider that full feed peer and then what we define as visible depends on the definition of our full feed peer.

Again, but my but my experience, if you change those by, like, twenty percent either direction, you'd probably end with about the same thing.

So, it's not I'm just curious.

So so the next thing was active measurements. So you're pinging stuff. What is the source, of the pings? You have multiple servers doing this?

We do have multiple provers, servers that are running this probing algorithm and we actually use, there's a research paper called a methodology called trinocular. I forget the title of the paper, but it's from, John Heidemann, USC.

And so basically this it's our approach is based on that. It's a little bit adapted, for our specific use case, but basically we're probing, something over four million slash twenty four blocks every ten minutes is what we do. And so what what it's basically doing is there's a set of target IP addresses that we kind of determine, this list of targets is determined in parallel to this, and then we're we're probing some set of these IP addresses every ten minute cycle.

And then depending on the responses or lack of that we get, we either classify this slash twenty four block as, up, which, you know, means it's connected, down or unknown, which kind of means, usually unknown means in the next cycle we need to do kind of a more comprehensive, measurement to make an actual determine on determination on whether or not it's up or down.

But then, yeah, again, similar to BGP, these slash twenty fours, they're mapped to the ASNs, the countries, and regions, and then we get kind of these aggregated statistics on what's the number of kind of slash twenty fours that we quote unquote saw during that ten minute.

And there's a, like, a refresh cycle or something of the target SOP?

Yeah. So, you know this is something our refresh cycle, it kind of depends on the input data sources but this is about quarterly.

And this is again just based on another, other research projects that do these full internet scans, and we're basically picking for each slash twenty four like what are the IP addresses that are most consistently responsive to, ICMP pings.

And, yeah, our provers we we are using ICMP but we actually had a a research paper last year, we're looking at kind of adapting some of the techniques from, like Zmap, you know, trying to add potentially TCP and UDP pings, because we've actually seen, there are some regions like in Africa, for example, where, some of the IP addresses are actually more likely to respond, to TCP or UDP, as a and some, like, Internet slash blocks, like, slash twenty four blocks that are, completely unresponsive to ICMP traffic.

And so Yeah.

I got we I've used a lot of ICMP in large scale in the past as well, and and I know I do get the, you'll hear the crystal. Like, it's no good because ICMP is blocked. I'm like, it it it is, but it it's not, like, Internet wide. So, like, if if you're measuring out a particular thing, then it might not work, but I don't know. I really haven't had a problem. There's also, like, the argument that, you know, the ICMP path is gonna take a different path than Traceroute. And that it there is instances when that's true.

On on the whole, it's kind of if you think about it, it's just, you know, the core of the Internet, it's just not worth it to spend a lot of time teasing apart, and and spending cycles thinking about, like, is this an ICMP packet or a TCP? Like, just just send it on. There there are once you start to get to into the, extremities of the Internet, then it starts to get more important, to make those decisions.

But I really haven't found that to be, you know, that that big of an obstacle, but there's a there's a kernel of truth there. But, anyway, so that's the that's the active measurement.

Yeah.

Ping. So I I say active measurement ping. Like, what what do you guys call it?

I guess, yeah, I call it active probing or a Active probing. We usually abbreviate it as.

I'll call it that from now on. Yeah. And so then, we've got, so we got two we got so the background radiation. So Yeah. Wanna talk about that?

Well, yeah, if you wanna talk about different names, that one goes by many names. Alright.

Yeah. I think, telescope is one of the most common ones, but, yeah, network telescope, internet background radiation.

And this one is basically, you know, you might even know more about what causes this traffic than I do, but, this is I know a little about it.

I know a little about it.

Oh, okay. But it's it's a unused, address space that basically is just monitoring incoming traffic. And so the idea here is that, all of the incoming traffic is is being generated somewhere else.

And I'm not familiar with all the research behind this, but there we do have some like anti spoofing heuristics just to make sure that these validating that the, you know, origin IP address on these packets is indeed the network that it's coming from. But we kind of use this like unsolicited traffic as a kinda another pulse of activity where there's there's signs of life so to speak. And, you know, kinda part of the reason with the telescope or background radiation analogy is that you're you're kind of unsolicited, you're just monitoring the traffic that's coming out of these different networks.

And so a similar thing here, you know, we're mapping these IP addresses to the locations and AS's, and we have various filters. We actually recently discovered there were, some networks that were doing vulnerability scanning. They were kind of throwing off our signals.

So we're identifying these subnets and removing them from our, visualizations. The, you know, the data is still there but, it doesn't really help when, you know, we're looking at connectivity for a country and we're seeing huge changes because there's a coordinated scan of this dark, IP space. But, yeah.

Okay. Yeah. I think, so my encounter you know, you mentioned the error spring in the beginning here, and that was also another as, you know, also an area where I cut my teeth early in Internet measurement.

And so we had this is back at we were still Renesas, but we had access to the DNS, some of the DNS query log stuff from Dyn.

And, and so they were still one thing that's interesting about Egypt when it went down, there was still traffic coming out of Egypt. It just couldn't get back in. And just so the routes were taken down, but the traffic could egress, a little like the Syrian outages that happen for student outages, you know, every year. Where, I know David Belson at Radars noticed this recently because they've got, you know, Cloudflare's got the big open resolve or the one to one to one, and then, you know, Dyna, you'd see this as well.

We'd see these spikes in DNS queries coming out of a place that supposedly is not is offline, but it's because at least in the case of DNS, you've got these retries, this retry storm that the queries can leave. And so then with Egypt, we saw it and I think some I think there was an academic paper that came out maybe from KEDA that also had documented this traffic, kinda unsolicited traffic was coming out. And this is basically oh, this isn't it's not even human driven. This is like this is like malware and spam and just all kinds of random if you have a large enough network, you're gonna have a whole bunch of crap just emanating thing trying to initiate connections.

There's misconfigurations or all kinds of things. And, so there's always like this noise. And so it's a it's a neat way to use that as a signal, because I like, you know, like I said, we we have the I have the benefit of having a bunch of traffic data, but that's really unusual. And so as a stand in, we just use DNS.

You guys using background radiation telescope, that usually is a good idea. It can give you although it's a bit of a noisy signal, I would say. You know, as a user, it's, it's something that you sometimes sometimes it lines up and sometimes, it's hard to know, what to make of that, as compared to the BGP and active probing.

For sure. Yeah. I the telescope signal is definitely the kind of noisiest of the bunch. It doesn't always have like a clear kind of, even with, you know, Google we might have a diurnal signal.

With the telescope one it can be really chaotic and for certain regions or countries where there's just not a lot of IP space, there's just not enough, kind of this background traffic that we, you know, we might not be able to get a strong signal for certain regions.

But at the same time we have found it like a very useful source for, you know, when we're looking at, you know, shutdowns versus some of the kind of what we'll call the spontaneous outages.

We we see that it's much more clear in the telescope. It like completely disappears like, so it's almost kind of like an on and off switch type signal.

And then there's other cases too where, you know, we've had instances where we're suspecting, government censorship, and maybe the, you know, it's a region where most of the population is accessing through like carrier grade NAT for example, and so we might actually see that the network stays online, the visible public IP addresses are still pingable, but then the telescope traffic just completely disappears during some suspected event.

That's not always the case but I think you know at the same time, you know telescope is definitely useful for, you know getting some signs of activity from users. But like I said, since we can't force the users to dial into this dark net telescope when we want them to, you know, we we can't prompt them to do it. So that that leads the signal to be a little bit inconsistent.

Sure.

And then, and then you've got Google search and that's just is that just country level not AS level? Like there's some I think there's some limitations on that right now.

So this is this is published as part of the Google transparency report, effort, and they publish it these numbers at the country level. I think each individual country has some anonymized like factor multiplication so it doesn't actually tell you like how many Google searches were run-in the country during that time period.

But you know this is one that is definitely useful, because it's such a popular service in a lot of regions that we get like a pretty strong and clear diurnal pattern in the usage. Sure. And you mentioned, you know, search, we actually in the kind of advanced view, we have the ability to pick other, signals from g t GTR.

This could be the Google Maps or Google Docs.

Sorry. What's g t r? What's g t r?

Sorry. G t r is Google Transparency Report.

That's our Oh, okay.

Got it. Abbreviation that we use.

That's what the kids call it.

Yeah. Well, it's when you're typing out API queries, it's g t r.

Got it.

Yeah. So the different products like sometimes, and this kind of goes back to what we were saying before that you know each of these signals can sometimes have a little bit of a a downside.

Or you know you need to use a little bit of context in order to figure out what's going on, but, you know, going one of the things that I first used the Google data for, is analyzing what was going on in during the invasion of Ukraine, and in that case there's so much going on that, like, you're really impact like, you're gonna change user behavior, like people have more important stuff to do than to run Google search queries at a consistent pattern compared to history. So we will actually see disruptions in this, compared to history. So we will actually see disruptions in this.

But Right. I think we see that we see that sometimes with an earthquake or some sort of other natural network in the immediate, like, there's a drop. Like, we'll see this in traffic. The traffic's dropped off.

Not necessarily because there's an outage. It's because everybody's doing not doing whatever they were doing a minute ago when the Yeah. World started shaking underneath their, their feet. They kinda stopped, and and that shows up, as a drop off in in traffic.

And then there's a there's a big spike as everybody goes to their phone to figure out what, what the heck does happened.

But, and all that is, like, human human driven.

Yeah. There's kind of a I mean, it's interesting but this is also, you know, terrible in a way. I always, I'm comfortable talking about, the invasion just because of all the terrible personal impact it would have had. But looking at the Google signal different product signals within Ukraine following the invasion, we did see, like, for example, Google Docs usage just completely drops off, in the days after. And it's like, well, people don't care about productivity.

And then, the Google Maps usage actually just spikes tremendously in the following days so that, you know, this is probably people trying to navigate out of this situation and, you know, maybe travel to find family or something.

But I I think it is in in a they're gonna ask you, in in IODA, can you can you choose which, Google, product is oh, okay.

It's hidden by default, but if you kind of use I think we're rethinking our approach to the the design here, but if you switch to the I the advanced view in IODA and click on like a expandable thing, I think, for the Google signal, it'll actually show all the different product signals. I think maybe I don't know. I guess I would guess about eight different signals, but I wouldn't quote me on that.

Yeah. There's maps, there's Gmail, there's photos, search. Because there's so many of them, that's why we use that drop down.

And we also saw Google Maps spike in Syria recently as as well. So we it is something we see with conflict. And the other usefulness or utility of having the Google product signal in there is if somebody is trying to understand if a application has been blocked or because YouTube actually is also in there. Or if if the Internet is down, that that can show you, you know, if all of the connectivity signals are up, but YouTube has completely dropped off. You know, in that case, it's it's a YouTube block.

It's it's nice that Google makes that, available. It's, you know, I I can't but think of the conversation I had a number I had attended a rights con, I think back in, I don't know, twenty fifteen or twenty sixteen. It was a long time ago. And there was someone from Facebook there, and I was chatting with this gentleman.

And I was like, how come Facebook doesn't, you know, contribute something? I think he give a give a talk on, you know, they had their way they're helping the world at the time. And and he's like he's like, he said, between you and me, like, people would be freaked out if they had any idea the level of data we have on, of just, like, you know, he said, we see, like, a building. Like, I live in he's like, I live in New York City.

I'll see a building go down. Like, the like, the they'll lose power. We know it. Like, we have the data on it.

Like, we have, you know, the location of all these people.

You have no idea. Like, who we you know, if you you could see how much they could see all the people with Facebook products on their phones being tracked twenty four seven by location.

You know, I try not to dwell on that too much. And I guess they just keep it to themselves and use it for their own purposes, but man, you know, we are modern society is a surveillance you know, environment. But, we'll see. Maybe we could talk about final minutes here. Just, what's what's what's coming new? You just announced some new features.

Well, I heard I p v six in there at some point.

Yeah. Right. Yeah. Yeah. Let's hear about that.

I'm actually I'm gonna let Amanda talk about our more immediate, things.

So as we mentioned with IODA, you can see connectivity signals at the country, region, or subnational, and network level. So we've always provided this, but what we've enabled is, the ability for users to see network signals localized to a specific country or region. So, previously, you might see an operator that operates in more than one country, and so you would see the signal for that network in both of those countries. And now you can, kind of localize or get more granular with that data. And we've already seen it be really useful and show a a different kind of severity and impact, looking at a network in a specific region versus just looking at that region or just looking at that network. So we're really excited about that. There's also a new way to navigate down to country, region, and network view so that if you pick a country, it automatically kind of filters the the networks that you can pick from.

And, actually, just yesterday or no. Just this morning, we we announced our new user resource hub. So Zach mentioned our our resource page, and we have tutorials, research papers, our glossary, all of the repositories and API information in there, and we'll be continuing to populate that with more, more of all of that as we continue to do research and, do reports on on outages.

And, Alright.

We're we're breaking news then. This is, you heard it here you heard it here first. Exactly. This is why you listen to Telemetry Now to find out the be the first one. You know, it's funny. I, when I was doing my analysis on the, Iberian Peninsula outage, I think that was the day when you guys rolled out the new thing. So while I'm working on this thing and, like, looking things up, I like, for, like, a second, I got, like, a dev link.

And so I I was like I was like, did did they, like, release, like, some and then and then and then I must have just, like, timed it perfectly. And then all of a sudden, I was like I was like, oh, there's new job tasks. You can combine search. They are.

This is super handy. You can you can look at an AS in a particular country and not just a I mean, my I think I think the way you guys started it was fine because I I don't know what the numbers are, but it's, you know, at least ninety percent of AS is really just live in one country, or a large you know, the vast majority, probably more than ninety percent. But there's that group that are, in multiple countries that, you know, the ones in a multiple country are often important. And, and so it's good to have that second filter to be able to zero in on the impact.

So that's that'll be very, very handy, but it was it was funny. I, like, I I I probably was the very first person running the queries live, because I was it was, like Excellent. It was going changing under my under my feet.

That's great.

Well, one of the nice things too is even within a country. So, you know, more specific regions. So for those ASs that do only operate in one country, you know, so long as we do have some thresholds in the back end that we only include them if it's, you know, above some threshold of like the number of IPs that we're monitoring but, you know, I haven't checked this, but we should be able to see, like, you know, what is Comcast status in this specific state, like California or something, is a view that you can check out.

Well, theoretically.

Assuming the g the geo. Like, what what how do you what do you what's your what's your geo, source?

Well, right now, we're, sponsored by IPInfo.

So we use them Oh, great.

In our back end. Yeah. And, you know, it's it's IP geolocation. That's all.

Yeah. Nobody's got, an error free, and and and an error free geolocation doesn't even exist because you've got some IPs are in multiple places and stuff.

So I I yeah. I've talked to some I think they're not bad.

I don't think they're not bad.

I would Yeah.

Yeah. I've talked to some operators that talk about, you know, IP assignments that just jump around a whole country. So it's there the, you know, the region subnational region level is not gonna be super accurate.

But, Yeah.

Or it can't be done. Like, so in, like, in my core I've I've come to find out that, you know, the the nation of Cuba and, Syria are both, like it's like a central d h DHCP server for, like, the whole country. Like, there's Yeah. You have no guarantee of what IP, is assigned in any part of the country, and therefore, that means you really can't do any geolocation or some assumption there these I this set of IPs are used in this place, and in in those countries it's it's centrally distributed. So there's I mean to my knowledge based on what I've heard, but I haven't I've never seen any reliable deal sub national geolocation in those places and some others.

Yeah. So we're also working on a couple other things, kind of in progress. We're hoping to rule them out over the next year or so.

You know first, we are actually working on integrating Mozilla telemetry data. This is just user reports basically of when they have issues, that you know that telemetry diagnostics that gets reported to Mozilla. They publish a summary of this based on, how much, how how many like connect connection issues, different air types that users are getting in different cities and countries around the globe. So currently, I think Amanda kinda hinted at this before, with some of the complications of different geography work, using different maps between the two. We have to kind of come up with a unified convention and convert from their city level data to our regions.

We're also adding some trace route measurements to this. We're hoping to get more vantage points up and running. And this is really inspired by, Doug's previous work, and blog posts from a long time ago, that I personally found very useful when I was doing internet research as a grad student.

Doug's blog articles and a lot of people at Renesas and Dine that contributed to this were tremendously useful in, helping kind of understand what was going on during some of these Internet events and, helping to identify what are the directions to look at. But, besides that, yeah, I mean we have a whole long list of things, a wish list of things that we would like to add and do but, you know, we're we are a academic institution project, our resources are limited and we kind of depend on what funding we can find and everything.

So, on that note, Zach and Amanda, thanks so much for joining today. It really has been a pleasure to learn more about the project and of course getting to know you as well.

So if folks want to learn more about the project, where can they find, where can they find information and, of course, the dashboard online?

Yeah. So we mentioned the dashboard. They can reach it by going to IODA dot live. We're also on Blue Sky and Mastodon.

If you search IODA, you'll probably find us. You'll see our graphs pointing out outages, on our on our livestream there. So those are the two places to find us.

Great. Thanks so much. And, to our audience, if you have an idea for an episode of Telemetry Now or you'd like to be a guest on the show, I'd love to hear from you. Please reach out at [email protected]. So for now, thanks so much for listening. Bye bye.