1

Problem

I have an express app running on an EC2 instance and use Node.js for my backend. I use Okta to mint/verify tokens for authentication, which has been working (untouched) for years. Suddenly, my applications no longer can access my server, and when I use postman to try and query some data I receive the following error: Error while resolving signing key for kid "Py9YS2tY1x--ESeMbRQVg6lWzKm45cQfZgk0GitmUpc"

Ive looked at the documentation for Okta but nothing seems to help. Again, this has been working for years. Here is the auth.js file doing the verifying:

const OktaJwtVerifier = require('@okta/jwt-verifier')

const oktaJwtVerifier = new OktaJwtVerifier({
  issuer: process.env.ISSUER,
  clientId: process.env.CLIENT_ID})

module.exports = async (req, res, next) => {
  try {
    const { authorization } = req.headers
    if (!authorization) throw new Error('You must send an Authorization header')

    const [authType, token] = authorization.trim().split(' ')
    if (authType !== 'Bearer') throw new Error('Expected a Bearer token')

    const { claims } = await oktaJwtVerifier.verifyAccessToken(token, 'api://default')
    if (!claims.scp.includes(process.env.SCOPE)) {
      throw new Error('Could not verify the proper scope')
    }
    next()
  } catch (error) {
    next(error.message)
  }
}

The issuer nor the clientID have been changed, nothing has changed whatsoever. The only thing I know of that has changed at all is that a little prior to this, the EC2 instance was upgraded from a t2.micro to a t2.small and the storage increased from 8 to 16 GB. This process does require the instance to be stopped/started/rebooted/etc. However, the stopping/starting/rebooting has been done multiple times before in the past and this issue has never arose.

I would greatly appreciate some help!

Additional Info

When expanding the storage size of the instance I followed the steps outlined in this answer to increase the actual partition size, and the steps outlined in this work-around to correct the 'unable to resolve host' error. Perhaps something was damaged or lost during the temporary storage of the drive during the partition grow?

What I found this morning was that if I run the server on my local and try to use Postman, instead of the 'resolving signing key for kid' error, the postman request simply hangs indefinitely, and I can see in my console that there was an uncaught exception thrown:

MongoDB connection error: MongoNetworkError: failed to connect to server [cluster-****-**-**.******.mongodb.net:*****] on first connect [MongoError: Authentication failed.
jcodes
  • 157
  • 1
  • 9
  • 1
    silly question, but is your requirement got reset after the upgrade maybe? – Philipp Grigoryev Mar 30 '23 at 12:01
  • @PhilippGrigoryev not a silly question, I am not familiar with all the components involved in this, as I inherited some of the code. What is the 'requirement' you mentioned? – jcodes Mar 30 '23 at 16:00
  • 2
    The root of your problem seems to be a network issue. Maybe DNS, maybe routes, etc. – jweyrich Mar 30 '23 at 17:08
  • @jweyrich I seem to have just corrected the issue by cleaning all of my node modules (npm cache clean) and rebooting the EC2 instance. My assumption is there was some cache the node module '@okta/jwt-verifier' was using that was corrupted/invalidated during growpart or instance IP change and was not corrected on first restart. However, I cannot validate or confirm. If you have any additional insight please add an answer and Im happy to approve. – jcodes Mar 30 '23 at 17:23
  • 1
    Ugh, autocorrection got me... Meant to say "environment" from where config params might be coming. Glad that you got it fixed! – Philipp Grigoryev Mar 31 '23 at 12:48
  • 1
    thanks @jweyrich, Netscope Client was causing my network issues in my case – Sam Sep 01 '23 at 07:52

1 Answers1

0

The issue appears to have come from some cache related to the '@okta/jwt-verifier' module, which was invalidated/corrupted during the partition grow or IP change. Rebooting the instance cleaned this and resolved the issue.

jcodes
  • 157
  • 1
  • 9