7

As far as I know the difference between map and imap is that map waits for all requests to finish and then returns the ordered data. Whereas imap returns data immediately and order less.

When I use:

urls = [...some_data...]
rs = (grequests.get(u,, hooks=dict(response=callback_f)) for u in urls)
r = grequests.map(rs)

the hook is used as soon as all requests finish, and the callback function is called.

When I use:

urls = [...some_data...]
rs = (grequests.get(u,, hooks=dict(response=callback_f)) for u in urls)
r = grequests.imap(rs)

then not a single request is sent.

According to the documentation map and imap have excatly the same API.

Is this the expected behavior? Should I not use hooks with imap? I am using Python 3.5.

max
  • 677
  • 1
  • 9
  • 34

1 Answers1

5

As far as I know the difference between map and imap is that map waits for all requests to finish and then returns the ordered data. Whereas imap returns data immediately and order less.

That's not really True. map does all requests immediatly and returns the result (this can take a while so that's why you probably said "waits for all requests to finish").

However imap returns a generator and does the requests only on demand. So you have to start iterating over the generator before requests are sent. Loop over the generator to get the results:

for single_request in r:
    # so something with "single_request"
sytech
  • 29,298
  • 3
  • 45
  • 86
MSeifert
  • 145,886
  • 38
  • 333
  • 352
  • So there is basically no way to make `imap` issue all the requests concurrently? – max Jan 31 '17 at 00:40
  • 2
    It has a parameter `size` to specify how many requests should be done concurrently. At least according to [`grequests.imap` on "Nullege Python Samples"](http://nullege.com/codes/search/grequests.imap) – MSeifert Jan 31 '17 at 00:45
  • @MSeifert also i use `.imap(..., size=1)` with huge requests so they are sent one after another in the background. – Winand Jul 27 '17 at 18:33
  • By default `size=2` in `imap` and `size=None` in `map` (uses this parameter for throttling) https://github.com/kennethreitz/grequests/blob/master/grequests.py – Winand Jul 27 '17 at 18:38
  • `each iteration only one request is made` is not exactly correct. The number of requests handled each iteration is not guaranteed, but will be less than or equal to the `size` argument. When the number of _results_ awaiting the reader is greater than `size`, the greenlets will block, awaiting the reader before making more requests. – sytech Apr 02 '20 at 03:56