1

There is some random website, http://www.example.com. This website use HTTP basic authentication.

I would like to make multiple requests to this site. But I don't want to login for each of the requests.

I have written the following code:

def loginWeb():
    global cookie

    homeURL = "https://www.example.com"
    request = urllib2.Request(homeURL)

    base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
    request.add_header("Authorization", "Basic %s" % base64string)   

    response = urllib2.urlopen(request)
    cookie = response.headers.get('Set-Cookie')

The aforementioned code fetches the cookie and uses it for subsequent requests.

The following code makes the subsequent request:

def getHTMLSourceCode(ID):        

    opener = urllib2.build_opener()
    opener.addheaders.append(('Cookie', cookie))

    response = opener.open('https://www.example.com/' + trID)
    sourceCode = response.read()

However, opener.open throws urllib2.HTTPError: HTTP Error 401: Unauthorized.

I also tried the following code, but this also throws the same error:

def getHTMLSourceCode(trID):        
    request = urllib2.Request("https://www.example.com/" + trID)       
    request.add_header('cookie', cookie)
    response = urllib2.urlopen(request)

    sourceCode = response.read()

    return sourceCode

urllib2.urlopen(request) throws urllib2.HTTPError: HTTP Error 401: Unauthorized.

By the way, I went through following answers, but the problem still persists.

Python urllib2, basic HTTP authentication, and tr.im

Keeping a session in python while making HTTP requests

python: urllib2 how to send cookie with urlopen request

Community
  • 1
  • 1
John Rambo
  • 906
  • 1
  • 17
  • 37

1 Answers1

0

You may want to try requests library and use it Session object

s = requests.Session()
s.auth = (username, password)
s.get(homeURL)

The cookie will be set to your session cookie s.cookies and you can use this session for other requests.

Nguyễn Thái Sơn
  • 291
  • 2
  • 8