Search Engines That Scan For Internet-Connected Services: Classification and Empirical Study

In this thesis, we revisit outdated definitions of Surface Web and Deep Web and provide new definitions and apply them to Internet search engines. We argue that the scope of the term "Web" is too narrow when referring to information on the Internet. We offer, and define, new terms to better describe the state of the Internet: Surface Internet, Shallow Internet, and Deep Internet. We use these terms to describe: Responding Internet-Connected Entity (RICE), Search Engine for Responding Internet-Connected Entities (SERICE), Web search engines, and Internet search engines. We explain how popular Internet-wide scanning services - Shodan and Censys - are SERICEs that index RICEs. In empirical work, we analyze scans from Shodan and Censys and determine they use few resources and provide an up-to-date view of the Internet.