Python is a popular scripting language developed by Guido van Rossum in 1991. It is highly readable, interactive, high-level, object-oriented, and interpreted. It typically uses English terms instead of punctuation and has lesser syntactic structures than other programming languages.

Some of the features of Python include:

  • It uses new lines to complete a command.
  • Python relies on white space, indentation, and defines the scope.
  • It is procedural, object-oriented, and functional.

In this article, we will dive deeper into some topics related to Internet access in Python. We will be discussing the Urllib.Request and Urlopen() functions present in Python, which help in accessing the Internet using Python.

Learn the Ins & Outs of Software Development

Caltech Coding BootcampExplore Program
Learn the Ins & Outs of Software Development

What Is Urllib?

In order to open URLs, we can use the urllib Python module. This Python module defines the classes and functions that help in the URL actions.

The urlopen() function provides a fairly simple interface. It is capable of retrieving URLs with a variety of protocols. It also has a little more complicated interface for dealing with typical scenarios, such as basic authentication, cookies, and proxies. Handlers and openers are objects that perform these services.

Python can also access and retrieve data from the internet, such as JSON, HTML, XML, and other formats. You can also operate directly with this data in Python. 

Fetching URLs With Urllib.Request With Syntax

We use urllib.request in the following way:

import urllib.request

with urllib.request.urlopen('<some url>/') as response:

html = response.read()

To temporarily store an URL resource in a location, we can use the tempfile.NamedTemporaryFile() and the shutil.copyfileobj() functions. 

Syntax

import shutil

import tempfile

import urllib.request

with urllib.request.urlopen('https://www.python.org/') as response:

    with tempfile.NamedTemporaryFile(delete=False) as tmp:

        shutil.copyfileobj(response, tmp)

with open(tmp.name) as html:

    pass

How to Open Url Using Urllib

After connecting to the Internet, import the urllib or the URL module.

Code

import urllib.request

webUrl=urllib.request.urlopen('https://www.python.org/')

print("result: "+str(webUrl.getCode()))

Output

result: 200 

Here, on running the code, if 200 is printed out as the result, that means that our HTTP request was successfully executed and processed, meaning our internet has worked fine.

The steps are highlighted below:

  • Import the urllib library.
  • Define the primary goal.
  • Declare the variable webUrl, then use the URL lib library's urlopen function.
  • The URL we're going to is www.python.org
  • After that, we are going to print the result code.
  • The getcode() function on the webUrl variable we had established is used to get the result code.
  • We'll convert it to a string so that it may be combined with our "result code" string.
  • This will be a standard HTTP code of "200," indicating that the request was properly handled.

Learn the Ins & Outs of Software Development

Caltech Coding BootcampExplore Program
Learn the Ins & Outs of Software Development

How to Read an HTML File for Your URL in Python?

By using the read() function in Python, we can read an HTML file in Python which will generate the HTML directly in the console.

Code (Python 3)

import urllib.request

webUrl=urllib.request.urlopen('https://www.python.org/')

print("result: "+str(webUrl.getCode()))

htmldata=webUrl.read()

print(htmldata)

Output

result: 200

<!DOCTYPE html>

<!--[if lt IE 7]>   <html class="no-js ie6 lt-ie7 lt-ie8 lt-ie9">   <![endif]-->

<!--[if IE 7]>      <html class="no-js ie7 lt-ie8 lt-ie9">          <![endif]-->

<!--[if IE 8]>      <html class="no-js ie8 lt-ie9">                 <![endif]-->

<!--[if gt IE 8]><!-->

<html class="js no-touch geolocation fontface generatedcontent svg formvalidation placeholder boxsizing retina flexslide" lang="en" dir="ltr" data-darkreader-mode="dynamic" data-darkreader-scheme="dark" style=""><script type="text/javascript" async="" src="https://ssl.google-analytics.com/ga.js"></script>

………….………….………….………….

  <li class="tier-2 element-4" role="treeitem"><a href="/about/help/" title="">Help</a></li>

  <li class="tier-2 element-5" role="treeitem"><a href="http://brochure.getpython.info/" title="">Python Brochure</a></li>

………….………….

………….

……

class="darkreader darkreader--sync" media="screen"></style><style type="text/css">#__wikibuy__ .__wikibuy.__onTop,#earny-root,#honeyContainer,#piggyWrapper,body~div:not(#gdx-bubble-host){position:absolute!important;z-index:100000!important}body[data-shop-url="https://www.honeybum.com"] header>.header{z-index:99999}.mm-slideout{z-index:auto}.sorry-for-this__empty-styles{position:relative;z-index:10000}</style><style class="darkreader darkreader--sync" media="screen"></style><div style="all: initial;"></div></div></body><grammarly-desktop-integration data-grammarly-shadow-root="true"></grammarly-desktop-integration></html>

Code (Python 2)

import urllib2

def main():

   webUrl = urllib2.urlopen("https://www.python.org/")

   print "result : " + str(webUrl.getcode()) 

   data = webUrl.read()

   print data

if __name__ == "__main__":

  main()

Output

result: 200

<!DOCTYPE html>

<!--[if lt IE 7]>   <html class="no-js ie6 lt-ie7 lt-ie8 lt-ie9">   <![endif]-->

<!--[if IE 7]>      <html class="no-js ie7 lt-ie8 lt-ie9">          <![endif]-->

<!--[if IE 8]>      <html class="no-js ie8 lt-ie9">                 <![endif]-->

<!--[if gt IE 8]><!-->

<html class="js no-touch geolocation fontface generatedcontent svg formvalidation placeholder boxsizing retina flexslide" lang="en" dir="ltr" data-darkreader-mode="dynamic" data-darkreader-scheme="dark" style=""><script type="text/javascript" async="" src="https://ssl.google-analytics.com/ga.js"></script>

………….………….………….………….

<li class="tier-2 element-4" role="treeitem"><a href="/about/help/" title="">Help</a></li>    

<li class="tier-2 element-5" role="treeitem"><a href="http://brochure.getpython.info/" title="">Python Brochure</a></li>

………….………….

………….

……

class="darkreader darkreader--sync" media="screen"></style><style type="text/css">#__wikibuy__ .__wikibuy.__onTop,#earny-root,#honeyContainer,#piggyWrapper,body~div:not(#gdx-bubble-host){position:absolute!important;z-index:100000!important}body[data-shop-url="https://www.honeybum.com"] header>.header{z-index:99999}.mm-slideout{z-index:auto}.sorry-for-this__empty-styles{position:relative;z-index:10000}</style><style class="darkreader darkreader--sync" media="screen"></style><div style="all: initial;"></div></div></body><grammarly-desktop-integration data-grammarly-shadow-root="true"></grammarly-desktop-integration></html>

The steps are highlighted below:

  • On the webURL variable, use the read() function.
  • The read variable allows you to read data files' contents.
  • Data is a variable that stores the complete content of the URL.
  • Run the code, and the data will be printed in HTML format.

Learn Python Development Online

To get internet access using Python and fetching data from different websites, we use the Urllib.Request and the urlopen() function are readily available in Python. To get more such information on Python and its various libraries, consider getting more in-depth with Python concepts. 

To get more resourceful knowledge on Mobile and Software development using Python, enrol in our world-class Post Graduate Program in Python Training, and get started with your Full Stack journey.

Our Software Development Courses Duration And Fees

Software Development Course typically range from a few weeks to several months, with fees varying based on program and institution.

Program NameDurationFees
Caltech Coding Bootcamp

Cohort Starts: 24 Feb, 2025

6 Months$ 8,000
Automation Test Engineer Masters Program

Cohort Starts: 8 Jan, 2025

8 months$ 1,499
Full Stack Java Developer Masters Program

Cohort Starts: 8 Jan, 2025

7 months$ 1,449
Full Stack (MERN Stack) Developer Masters Program

Cohort Starts: 8 Jan, 2025

6 Months$ 1,449

Recommended Reads