Proper way to use page.waitForNavigation in Puppeteer

Question

I'm a bit confused about page.waitForNavigation. I know what it is and what it does but I get different results based on the interenet speed (that's what I think is the factor).

Imagine this code:

await page.$eval('#username', (el , setting ) => el.value = setting.username , setting );
await page.$eval('#password', (el , setting ) => el.value = setting.password , setting );
await page.$eval('form', form => form.submit());
await page.waitForNavigation();
await page.goto( 'http://example.com/dashboard'  );

It fills the login form, submits the login form, waits for the form to be submited and then redirects to dashboard.

This works fine on my localhost which has slower internet speed (compared to the server), but when I upload it to the server, I get

Error: Navigation Timeout Exceeded: 30000ms exceeded

On the server it works fine if I remove await page.waitForNavigation(); from the code and I get redirected to dashboard.

But now on localhost, I get redirected to the dashboard before the form can be submitted. I get you cant see dashboard , your not logged in or something like this.

I think the deciding factor is the speed of internet.

On the server, I have a very high speed so the form gets instantly submitted and it's done before the await page.waitForNavigation() line so I get a nvigation timeout error.

But on the localhost with the slower speed, the form needs more time to be submitted so I need to have await page.waitForNavigation() after submitting the form, otherwise I get redirected to the dashboard before the form has any chance to be submitted.

I'm looking for advice from someone with more experience working with Puppeteer on how to handle this kind of situation. Right now, I keep editing my code when running on server or localhost, which works, but it's very annoying!

After using

async function open_tab(setting) {
  const page = await global_browser.newPage();
  await page.setViewport({
    width: 1000,
    height: 768
  });

  return await new Promise(async(resolve, reject) => {
      await page.$eval('#username', (el, setting) => el.value = setting.username, setting);
      await page.$eval('#password', (el, setting) => el.value = setting.password, setting);
      await Promise.all(
        page.$eval('form', form => form.submit()),
        page.waitForNavigation()
      )
      await page.goto('http://example.com/dashboard');
      resolve();
    }).then(() => {
      console.log(' -> don!  ');
      page.close();
    })
    .catch(() => {
      console.log(' -> somethign went wrong !');
      page.close();
    })
}

I get

(node:14812) UnhandledPromiseRejectionWarning: TypeError: undefined is not a function
    at Function.all (<anonymous>)
    at Promise (D:\wamp\www\gatewayCard\robot\server.js:287:23)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
(node:14812) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
(node:14812) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

use try...catch instead of such `new Promise` block when you are using async...await. — Md. Abu Taher, Sep 03 '18 at 11:39
btw why are you filling the form without going to the website first? XD — Md. Abu Taher, Sep 03 '18 at 11:47
@Md.AbuTaher i've deleted extra codes like going to website so we can focus on the problem — hretic, Sep 03 '18 at 13:52
@Md.AbuTaher `mew Promise` was suggested `https://stackoverflow.com/questions/52059034` here bcuz i needed a timeout for function — hretic, Sep 03 '18 at 13:52
Maybe the code you removed are causing the problem. ;) Check out my answer, and go thru the links provided. You will have a better understanding of how async await works :) — Md. Abu Taher, Sep 03 '18 at 14:07
@Md.AbuTaher thanx , the reason for using promise inside async function was i really had to have a timeout for async function to avoid the browser being stuck somewhere .... i needed timeout for my async function to close the tab in that case and apparently using promise is the way to do it — hretic, Sep 03 '18 at 17:15
You can learn more about Promise.race :) , from what I see, you should definitely try to study these cause all of your recent problems are quite similar and I admire your courage to post questions for all these. — Md. Abu Taher, Sep 03 '18 at 18:09

score 20 · Accepted Answer · edited Aug 09 '19 at 14:28

It happens because the navigation might happen on submit before you wait for it.

Use the submission and waitForNavigation inside one promise using Promise.all, so it will wait for both of them instead of one at a time.

await Promise.all([
 page.waitForNavigation(),
 page.$eval('form', form => form.submit())
])

or,

await Promise.all([
 page.$eval('form', form => form.submit()),
 page.waitForNavigation()
])

either should work.

EDIT 1:

The edit is totally offtopic to your main question. You are not using a proper async...await in your code. Here is a better code.

The beauty of async...await functions are you can do less code and better readability and maintainability. It's a double-edge sword, but worth it if used properly.

async function open_tab(setting) {
  try {
    const page = await global_browser.newPage();
    await page.setViewport({
      width: 1000,
      height: 768
    });
    await page.goto('http://example.com/dashboard');
    await page.$eval('#username', (el, setting) => el.value = setting.username, setting);
    await page.$eval('#password', (el, setting) => el.value = setting.password, setting);
    await Promise.all(page.$eval('form', form => form.submit()), page.waitForNavigation());
    console.log(' -> don!  ');
    await page.close();
  } catch (error) {
    console.log(' -> somethign went wrong !', error);
    await page.close();
  }
}

EDIT 2:

Here in your code,

return await new Promise(async (resolve, reject ) => {

this is wrong on many steps. async functions returns a promise anyway, so you are returning a promise inside a promise without catching it and using await with it. Check out your code early otherwise you will face huge problems soon.

It seems you should learn a bit more about async await first. Here are some helpful links for you, these will explain timeouts and everything you want to learn.

hi , seems logical but when i updated my code i got an error .. pleas look at my question i've updated with error — hretic, Sep 02 '18 at 13:51
whats on this line? (D:\wamp\www\gatewayCard\robot\server.js:287:23) — Md. Abu Taher, Sep 03 '18 at 02:05
i've updated my code so there is more context .... promise.all is inside another promiseso maybe thats what cuzing error — hretic, Sep 03 '18 at 08:06

Proper way to use page.waitForNavigation in Puppeteer

1 Answers1