Anatomy of a Web Request

Table of Contents

    Subscribe for Email Updates
    Web server request

    When you click on a link or type a website address into your browser's address field the invisible gears of the Internet get to work. When everything is working hunky dunky you probably give no thought to how it all works. But, if you're like most people, you become lost and helpless when a single piece of this massive orchestration fails. My goal is to help clear that up for you, at least a bit.

    Components of a Web Request

    In order for everything to work the following three pieces must work together.

    1. domain registration - a domain has to be properly registered
    2. initial domain lookup - your computer or device needs to know where the web server is that will process your request
    3. web server processing - the web server has to understand what you want

    Web Request Component #1: Domain Registration

    It all starts with registering a domain. Notice I wrote "registering" and not "buying". You don't own a domain, you register the use of a domain for 1-10 years at a time. So the next time someone says, "Oh, I own that domain," you can pretentiously correct them and say "Wrong sir, you registered the domain. You do not own the domain." Not a good way to make friends but you can sound really smart.

    In reality, the quasi-owner of the domain is ICANN (Internet Corporation for Assigned Names and Numbers). They are the organization that allows other companies to become a top level domain registry, who can then sell domains using the licensed top level domain.

    SIDE NOTE:
    A top level domain (or TLD) is the .com, .net, .us, or .edu part of the domain. 

    A couple years ago there were only a handful of TLDs, such as, .com, .org, .net, .us, .cc, .gov, .edu, etc. Now there are hundreds of top level domains and more arriving regularly. You can register yourdomain.me, yourstore.blackfriday, or even आपकी साइट.भारत  (Hindi for yoursite.bharat). 

    Here's how it works:

    Domain Registration

    1. A user, or registrant, visits a registrar like GoDaddy, Network Solutions,  enom, or one of the other thousand ICANN accredited registrars around the world. 
    2. Once the registrar knows what domain you want it then contacts the TLD's registry operator. For .com TLDs the registry operator is Verisign. The registry then checks if the TLD is available and returns the status back to the registrar.
    3. If the domain is available, the registrar notifies the user and the user pays the fee for a registration of 1-10 years. Some domains require additional information and the registrar is responsible for obtaining that information or verifying proper intended use of the domain. An example is the .us domain. Organizations that register a .us domain must be a citizen, resident, organization, or foreign entity with a US presence.

      Also during the final registration process, or during a subsequent update to the domain, the user can the name servers that will handle its requests (more on this later).

    After registration, the registrar (or an associated registrar reseller) is responsible for notifying customers when domains are about to expire and also for maintaining current contact information for the domain.

    IMPORTANT!
    If you, or someone in your organization is not responsible for keeping your domain registered then you need to do this ASAP. We've had several customers who ignored expiration notices from their registrar and lost their domain. One even had a foreign entity register the domain as soon as it was available, create an exact copy of their website, and change all links on the site to point to an e-commerce site for Gucci watches. And they were prepared to do this because they copied their entire website before the domain expired. I can only imagine what important emails were also being routed to them!

    Web Request Component #2: Domain Lookups

    A domain is the friendly representation of a more complicated numbering system called an IP address (or network address). You should be quite familiar with domains. They are used to find websites, send emails, and connect computers to each other for processing all sorts of data (credit cards, file storage, video streaming, etc). marketpath.com is our domain that translates into the IP address 65.52.245.115. Rather than typing in that long number I simply type marketpath.com. That's when the magic begins.

    Domain Lookup

    1. The user types marketpath.com into the browser's address field. The computer contacts its assigned domain name system (DNS) to get the IP address of the web server hosting Marketpath's website.
    2. The DNS server, if it hasn't looked up the domain recently, will reach out to the Internet's root name server to figure out where to go next. The root name servers holds the registry locations of all the TLD's, such as, .com, .net, and .org. Basically, the root name server tells the DNS server where to search next, or for .com domains, who has all the information about the .com domains.
    3. The DNS server then contacts the registry server for the .com domain. This server will tell the user's DNS server where marketpath.com holds all their records. Verisign is curerntly the contracted agency in charge of maintaining the .com TLD
    4. The DNS server then goes to the marketpath.com name server. This server answers all requests for marketpath.com and it's sub-domains (e.g. www.marketpath.com, mail.marketpath.com, etc). We use a reliable, hosted DNS provider called Zerigo to handle all of these requests. If Zerigo finds the domain or sub-domain, it will then send the destination IP address for marketpath.com - i.e. 65.52.245.115.
    5. The DNS server relays the resulting IP address back to the user's browser. The browser stops communication with the DNS server for that domain and redirects its attention to the destination web server for marketpath.com.

    All of this happens in less than a second and once the first lookup takes place your computer and local DNS server will store the results in a cache so it doesn't have to perform the lookup every time.

    SIDE NOTE:
    This is often where delays occur when seeing a newly launched website. Your website agency or team says "Hey, the new site is live! Congrats!" but you may not see it for a while because your old website's IP address is cached in your browser, on your computer or in the DNS chain somewhere down the line. When your web development team tells you it could be between 1 - 48 hours before you see your new site they really mean it. Generally, there's a time-to-live (TTL) value stored with each DNS record that tells devices how long to maintain the cached value before looking it up again. Not all DNS servers adhere to that, however.

    Web Request Component #3: Web Server Request

    Finally, we get to the web server which will eventually send you a web page with text and pretty pictures. If you're still here reading, then kudos to you. You've survived the onslaught of technical drabble that summarizes how we get to the final web server that has the website for marketpath.com. In reality, there are many more layers to this entire process where something can go wrong and cause it all to fail. Redundancy is the magic word and well thought out systems, including websites, have built-in redundancies and fail-overs that help mitigate any single point of failure.

    The web server side is where those of us at Marketpath spend most of our time. We build marketing websites and custom web applications, including e-commerce storefronts, customer portals, and more. 15-20 years ago most websites consisted primarily of static files, that is, files that aren't processed but just sent straight back to the users web browser. Static files include html files (i.e. the code that defines the layout of a page), images, videos, style sheets, and JavaScript.

    Most experienced website developers build sites with some sort of dynamic capability. There are few websites with just static elements. Instead, sites more often use a content management system, like Wordpress, Marketpath CMS, Joomla, or Sitefinity, to name just a few. These systems provide easy-to-use tools for editing and publishing content to the website. Typically there is no need to know code. But the systems themselves are still built on top of a general web server architecture like Microsoft's IIS (Internet Information Services), Apache, or Node.js. They all answer and process web requests in similar ways. One common process is shown below. 

    Web server request

     

    1. The user's browser sends the initial request to the web server asking for a single resource. If I type in "marketpath.com" I will implicitly be requesting the home page (i.e. http://www.marketpath.com/home). 
    2. The web server has a service that is always listening for requests and captures that request. 
    3. The service uses its route handler to decide if it needs to return a static file or send this to a script (dynamic code) for further processing. 
    4. If the user is requesting a static file, it will grab the file and return it. If we're asking for the home page it is likely to be dynamic, and if so, it will move down the chain and begin processing the script.
    5. The script often uses common object libraries to retrieve information. For example, an e-commerce site may use it's store library and specifically request a product object.
    6. The requested object may then connect to the database where all the data is stored and return information for that product. 
    7. Eventually, the completed request will send the data, whether static or dynamic, back to the web service.
    8. The web server will send the content back to the user's web browser and begin rendering the html content. Starting at the top, it will look for additional resources that need to be loaded.
    9. Additional requests to each resource in the html markup will load in order - typically (if the web developer is experienced) style sheets will load first, then images, and finally Javascript files.
    10. Requests back to the web server are made for each additional resource.
    11. The browser handles each resource properly. It renders styles in the style sheet, displays images, and processes Javascript.

    And that's it! Easy, right? Ok, I'll admit, there's a lot going on here. And this is just one example. Web server processing methods can vary dramatically from this flow chart. However, this is a fairly common flow. Just keep in mind that when your site has problems and your developer or IT crew says "We're looking into it" you should be patient and understand there are a ton of places where something can go awry, many often out of your team's direct control.

    Related Tags:

    About the Author

    Matt Zentz

    Matt Zentz launched Marketpath from a small Broad Ripple bungalow in February 2001 with a focus on custom web application development. He built the first, basic version of a hosted CMS called Webtools and shortly afterward expanded his team and created the first version of Marketpath CMS.

    Matt has worked for a national consulting firm, taught computer programming to high school juniors and seniors , and led the information technology arm of the auxiliary business units at Indiana University.

    Matt graduated from Indiana University in 1999 with a B.S. in Computer Science and has built custom web applications since 1995. Matt is husband to an amazing & supportive wife, has three beautiful children, supreme master to Archimedes (Archie) the dog, and mostly tolerant victim of 2 flying rats (cockateils).

    He coaches various kid sports, enjoys furniture and home renovation projects, and plays guitar and piano. Matt is also active with his church as a parishioner, technical advisor and board member on the festival committee.

    Subscribe for Email Updates