About

Regional Internet Registries (RIRs) publish WHOIS, route object delegation (in Statistics Exchange files), and reverse DNS (rDNS) zone files. These are are valuable resources for networking research and engineers yet they contain inconsistencies and are not all available long-term. In this work, we consolidate and make available longitudinal RIR-level data, aiming to lower the barriers to start working with these data.

How-To

We host the consolidated rDNS and WHOIS data in an S3-compatible Object Storage. As such, you can load the data directly from tools such as Apache Spark. Alternatively, you can download the data and process them as you see fit. To get you started, we created a basic Jupyter Python notebook for inspiration here.

Consolidated WHOIS data

We store the consolidated WHOIS data in a tiered (year, month, day) hierarchy, which popular tools for data engineering can use for partition discovery as well as optimisation. The partitioned data contains per record information such as the source RIR, WHOIS serial number, object created and last-modified dates.

{"serial": 748705, "use_route": true, 
"prefixes": ["23.219.183.0/24"],"start_address": "23.219.183.0", 
"end_address": "23.219.183.255", "descr": "Akamai Technologies", 
"origin": 20940, "mnt-by": "MNT-AKAMAI", 
"source": "ARIN", "created": 1555027200, 
"last-modified": 1555027200, "status": "ALLOCATED",
"netname": null, "country": "US", "af": 4}
{"prefixes": ["23.219.0.0/16"], "start_address": "23.219.0.0", 
"end_address": "23.219.255.255", "rfc_2317": false, 
"timestamp": 1684357200, "source": "ARIN", "af": 4, 
"rdns": {"name": ["219.23.in-addr.arpa."], 
"origin": ["23.in-addr.arpa."], "ttl": 86400, 
"rdclass": "IN", "rdatasets": 
{"NS": ["ns{1-8}.reverse.deploy.akamaitechnologies.com."]}}}
Consolidated rDNS data

We enrich the data, e.g., by adding a classless delegation flag. The prefixes in RIR-level zones largely follow octet boundaries, but CNAME are sometimes present for classless delegation, i.e., RFC 2317. We store the consolidated rDNS data in a tiered hierarchy similar to the WHOIS data and key records in the same manner.

Team Members

Alfred Arouna
PhD Student
alfred[at]simula[dot]no

Ioana Livadariu
Research Scientist
ioana[at]simula[dot]no

Mattijs Jonker
Assistant Professor
m.jonker[at]utwente[dot]nl
Data Schema
Field Datatype Description Dataset
serial INTEGER Internal serial number for published WHOIS WHOIS
prefixes ARRAY of STRING Allocated prefixes or delegated rDNS Both