Web Crawling:-

               Search engine works by matching the required information from different websites and storing many web pages. These pages are then retrieved by a Web crawler ( also known as a spider).

              The contents of each page are then analyzed to check how it should be indexed. Data collected on web pages is stored in an index database for use in later queries. The purpose of an index is to get information as quickly as possible.