Use asyncio to login multiple users to a server

Question

I have a server which I need to login many users to. During each login the server has to do lots of things that take a while like make requests to other servers/databases and wait for them. So whilst one user is logging in I would like the other users to start logging in too.

I have tried to figure out how the asyncio module works but have not found any code that does what I want it to do.

Note: I am using python 3.6 so some of the asyncio features are different to 3.7.

import asyncio
import requests

class user:
    def __init__(self, port):
        # Create user here
        self.port=port
        self.data = requests.get(someURL,port)

    async def login(self,email):
        # Step 1 of logging in
        requsts.post(someURL, self.port, email) # Logs them in but takes a while to process everything
        while True:
            data = requests.get(someURL, self.port)# Gets data from server
            if data.connected == True:
                break
            else:
                pass
                #Go onto logging in next user while we wait for the server to do things

        # Step 2 of logging in
        requests.post(someURL, self.port, email)
        while True:
            data = requests.get(someURL, self.port)
            if data.loggedIn == True:
                break
            else:
                pass
                #Go onto logging in next user while we wait for the server to do things

listOfUsers = []
for i in range(3):
    listOfUsers.append(user(3000+i)) # Creates 3 users

async def loginListOfUsers():

    for user in users:
        await user.login(user.port + "@gmail.com") # Logs in 3 users with i@gmail.com


loop = asyncio.get_event_loop()
loop.run_until_complete(loginListOfUsers())

I would like to know:

1. Is asyncio the right tool for what I am trying to do

2. How I would use asyncio to do it

Here is what I want/think asyncio can do:

Create an event loop,

Add coroutines to this loop and run them,

When it gets to a certain point e.g. an await statement it stops running that coroutine and moves on to the next one in the loop, pushing that one to the back of the loop

I don't have a great understanding of asyncio so am probaby very mistaken but hopefully you can understand what the problem I have is.

Answer 1

When it gets to a certain point e.g. an await statement it stops running that coroutine and moves on to the next one in the loop, pushing that one to the back of the loop

That is exactly what asyncio is good at. To execute tasks in parallel, your code needs to satisfy two pre-requisites:

1. It needs to actually be async;

2. It needs to request the coroutines to run in parallel and not await them in series.

The first point becomes clear if you take a careful look at the content of coroutines from the question, such as user.login. Although they technically are coroutines (they all begin with async def), they don't await anything, so they never offer asyncio a chance to suspend their execution to do something else.

To make them useful you need to replace every invocation of blocking code with a non-blocking equivalent. The simplest bad-aid to do that while still using requests is to use run_in_executor to postpone execution to another thread. In other words, replace:

data = requests.get(someURL, self.port)

with

data = await loop.run_in_executor(None, requests.get, someURL, self.port)

The second point can be met by a simple change to loginListOfUsers. Rather than awaiting each coroutine in series (which basically ensures that they will not run in parallel), it needs to start them all in advance and wait for them to complete, for example using asyncio.gather:

async def loginListOfUsers():
    # assemble a list of coroutines without actually running them
    todo = [user.login(user.port + "@gmail.com")
            for user in users]
    # now let all the coroutines run in parallel,
    # and wait for them to complete
    await asyncio.gather(*todo)

Note that using run_in_executor will use threads under the hood, and will only allow as many parallel connections as there are workers in the thread pool. To make better use of asyncio, you can switch to aiohttp which supports multiple connections without each being backed by an OS thread.

Answer 2

There are two kinds of libraries in python. Async-libraries and blocking-libraries. You should try only to use asyncio-libraries when you use asyncio.

requests is a blocking libarie. Here is an example, how to use it with asyncio: https://stackoverflow.com/a/23784727/4493241

I prefer not to use blocking-libraries at all. So I would recommend to rewrite your code without request. Here is a http-client example with asyncio: https://asyncio.readthedocs.io/en/latest/http_client.html