1

I already spent several hours googling around lot of sites but I can't maange to fix i. Let's hope someone can help me here:

LAPTOP (the file gets downloaded succesfully without issues):

wget -d http://EXAMPLE.COM:80/movie/USERNAME/PASSWORD/video.mkv
DEBUG output created by Wget 1.17.1 on linux-gnu.

Reading HSTS entries from /home/mylinuxuser/.wget-hsts
URI encoding = ‘UTF-8’
--2018-08-03 01:30:35--  http://EXAMPLE.COM/movie/USERNAME/PASSWORD/video.mkv
Resolving EXAMPLE.COM (EXAMPLE.COM)... SOME_IP_ADDRESS, DIFFERENT_IP_ADDRESS
Caching EXAMPLE.COM => SOME_IP_ADDRESS DIFFERENT_IP_ADDRESS
Connecting to EXAMPLE.COM (EXAMPLE.COM)|SOME_IP_ADDRESS|:80... connected.
Created socket 3.
Releasing 0x0000558aa8dbba40 (new refcount 1).

---request begin---
GET /movie/USERNAME/PASSWORD/video.mkv HTTP/1.1
User-Agent: Wget/1.17.1 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: EXAMPLE.COM
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 302 Found
Server: nginx/1.14.0
Date: Fri, 03 Aug 2018 00:14:42 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *
Location: http://ANOTHER_IP_ADDRESS:80/movie/USERNAME/PASSWORD/video.mkv?token=SxoOVkYKEF4WAAcAU1ZTBlBRUFMKBgBUUlFSBQgHAQkBClNVAVoEAVYXGkEVQRYGAAtuCFdHCgddV1wGHEFESlVKOV5RQAhGDA0OUVMDRk9DElgMVkcKB1BSVwUGUAIAAxRER1wGEF4WBgdXXgNGT0MDSRVWF15XCT4AUkYKUlwSAkQVGUBdCmtRUw4HWwBBW0QBQx9HWUUVQ14VcwJDSVhXCFIVNVMWUV1ZFhVQRCETCVAFUQReUkUyAUVGClJcQxpKFVcLRhZVQVNBXBdWUFBXE00RBl9DCxUWThJZE35yGkoVUBpGAVpGXwwIF15BDA1HQx9HWUU6EwFERBFUWF1dFBUPQAJGGBdbAh5qBwwPCFQCRwxfWBZDXhUBQR0bXVcIXkENQDtEXFJBXFsRDw0b

---response end---
302 Found
Registered socket 3 for persistent reuse.
URI content encoding = ‘UTF-8’
Location: http://ANOTHER_IP_ADDRESS:80/movie/USERNAME/PASSWORD/video.mkv?token=SxoOVkYKEF4WAAcAU1ZTBlBRUFMKBgBUUlFSBQgHAQkBClNVAVoEAVYXGkEVQRYGAAtuCFdHCgddV1wGHEFESlVKOV5RQAhGDA0OUVMDRk9DElgMVkcKB1BSVwUGUAIAAxRER1wGEF4WBgdXXgNGT0MDSRVWF15XCT4AUkYKUlwSAkQVGUBdCmtRUw4HWwBBW0QBQx9HWUUVQ14VcwJDSVhXCFIVNVMWUV1ZFhVQRCETCVAFUQReUkUyAUVGClJcQxpKFVcLRhZVQVNBXBdWUFBXE00RBl9DCxUWThJZE35yGkoVUBpGAVpGXwwIF15BDA1HQx9HWUU6EwFERBFUWF1dFBUPQAJGGBdbAh5qBwwPCFQCRwxfWBZDXhUBQR0bXVcIXkENQDtEXFJBXFsRDw0b [following]
] done.
URI content encoding = None
--2018-08-03 01:30:36--  http://ANOTHER_IP_ADDRESS/movie/USERNAME/PASSWORD/video.mkv?token=SxoOVkYKEF4WAAcAU1ZTBlBRUFMKBgBUUlFSBQgHAQkBClNVAVoEAVYXGkEVQRYGAAtuCFdHCgddV1wGHEFESlVKOV5RQAhGDA0OUVMDRk9DElgMVkcKB1BSVwUGUAIAAxRER1wGEF4WBgdXXgNGT0MDSRVWF15XCT4AUkYKUlwSAkQVGUBdCmtRUw4HWwBBW0QBQx9HWUUVQ14VcwJDSVhXCFIVNVMWUV1ZFhVQRCETCVAFUQReUkUyAUVGClJcQxpKFVcLRhZVQVNBXBdWUFBXE00RBl9DCxUWThJZE35yGkoVUBpGAVpGXwwIF15BDA1HQx9HWUU6EwFERBFUWF1dFBUPQAJGGBdbAh5qBwwPCFQCRwxfWBZDXhUBQR0bXVcIXkENQDtEXFJBXFsRDw0b
conaddr is: SOME_IP_ADDRESS
Releasing 0x0000558aa8dbe590 (new refcount 0).
Deleting unused 0x0000558aa8dbe590.
Connecting to ANOTHER_IP_ADDRESS:80... connected.
Created socket 4.
Releasing 0x0000558aa8dbe590 (new refcount 0).
Deleting unused 0x0000558aa8dbe590.

---request begin---
GET /movie/USERNAME/PASSWORD/video.mkv?token=SxoOVkYKEF4WAAcAU1ZTBlBRUFMKBgBUUlFSBQgHAQkBClNVAVoEAVYXGkEVQRYGAAtuCFdHCgddV1wGHEFESlVKOV5RQAhGDA0OUVMDRk9DElgMVkcKB1BSVwUGUAIAAxRER1wGEF4WBgdXXgNGT0MDSRVWF15XCT4AUkYKUlwSAkQVGUBdCmtRUw4HWwBBW0QBQx9HWUUVQ14VcwJDSVhXCFIVNVMWUV1ZFhVQRCETCVAFUQReUkUyAUVGClJcQxpKFVcLRhZVQVNBXBdWUFBXE00RBl9DCxUWThJZE35yGkoVUBpGAVpGXwwIF15BDA1HQx9HWUU6EwFERBFUWF1dFBUPQAJGGBdbAh5qBwwPCFQCRwxfWBZDXhUBQR0bXVcIXkENQDtEXFJBXFsRDw0b HTTP/1.1
User-Agent: Wget/1.17.1 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: ANOTHER_IP_ADDRESS
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 03 Aug 2018 00:29:53 GMT
Content-Type: video/x-matroska
Content-Length: 2071078584
Connection: keep-alive
Accept-Ranges: 0-2071078584
Content-Range: bytes 0-2071078583/2071078584

---response end---
200 OK
Disabling further reuse of socket 3.
Closed fd 3
Registered socket 4 for persistent reuse.
Length: 2071078584 (1.9G) [video/x-matroska]
Saving to: ‘video.mkv’

VPS (it doesn't download anything as it can be seen below):

wget -d http://EXAMPLE.COM:80/movie/USERNAME/PASSWORD/video.mkv
DEBUG output created by Wget 1.19.4 on linux-gnu.

Reading HSTS entries from /root/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'video.mkv' (UTF-8) -> 'video.mkv' (UTF-8)
--2018-08-03 00:35:19--  http://EXAMPLE.COM/movie/USERNAME/PASSWORD/video.mkv
Resolving EXAMPLE.COM (EXAMPLE.COM)... SOME_IP_ADDRESS, DIFFERENT_IP_ADDRESS
Caching EXAMPLE.COM => SOME_IP_ADDRESS DIFFERENT_IP_ADDRESS
Connecting to EXAMPLE.COM (EXAMPLE.COM)|SOME_IP_ADDRESS|:80... connected.
Created socket 3.
Releasing 0x0000561201148270 (new refcount 1).

---request begin---
GET /movie/USERNAME/PASSWORD/video.mkv HTTP/1.1
User-Agent: Wget/1.19.4 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: EXAMPLE.COM
Connection: Keep-Alive
Referer: EXAMPLE.COM

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 401 Unauthorized
Server: nginx/1.14.0
Date: Fri, 03 Aug 2018 00:19:26 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *

---response end---
401 Unauthorized
Registered socket 3 for persistent reuse.
] done.

Username/Password Authentication Failed.

Does anyone know what command should I run in the VPS or what else I should install so it can download the file?

Juan Martin
  • 67
  • 6
  • Looks like user and password might not be the correct credentials when running from VPS. No way to check that out, most evidence has been string-replaced. – alariva Aug 03 '18 at 02:56
  • I run exactly the same command from my laptop linux terminal than from the vps... I'm guessing it might due to some missing packages that need to be installed in the VPS, or due to differences between wget versions (the one from the VPS is newer). I got very very basic linux knowledge but to me it looks like the VPS fails to get the token, while the linux laptop terminal doesn't have any issue with that. But I really don't know if that could be the cause and how to resolve it – Juan Martin Aug 03 '18 at 09:32
  • Then trying to discard options over version and software might help. Have you tried to downgrade wget in your vps? Or upgrade in your laptop to reproduce the problem? Before that you may also test result with aria2 tool as a replacement for wget. Think of discarding scenarios and lets build a soft, version, env table with recorded results. – alariva Aug 03 '18 at 14:29
  • Another possible check is the .wget-hsts contents. Have you tried using the same in your laptop from your VPS? And another try is with --no-hsts param from VPS. – alariva Aug 03 '18 at 14:35
  • You might also want to check the following post to make sure you keep the session data after the first redirect (http 302) https://stackoverflow.com/questions/1324421/how-to-get-past-the-login-page-with-wget – alariva Aug 03 '18 at 14:50
  • The issue hasn't been resolved yet. Now I got two different VPS servers (from different providers) both using Ubuntu 16.04 LTS and both using wget 1.17.1 One of them can download the files succesfuly but the other one fails in the authentication just as I mentioned in my opening post – Juan Martin Aug 08 '18 at 16:03
  • Despite the issue hasn't been resolved, you didn't post any updates regarding the problem diagnose. You now even have 2 different environments with different behaviors. It's mostly probable that it's a versioning problem (wget or any other platform problem). So we can narrow down the problem by comparing the three versions of wget and OS you have and the expected results. Would you help out building this table? I have posted [this pastebin](https://pastebin.com/wxHrB61Q) as per my undestandig, but feel free to fork and edit as needed. – alariva Aug 08 '18 at 16:52
  • Hello. I can't pick any file called .wget-hsts despite the logs I posted mention something was done there. In one of the computers I could but it was empty. Anyway here is an update trying to diagnose the problem: https://pastebin.com/EyrkYAZH PS: I no longer have the VPS 2 – Juan Martin Aug 08 '18 at 23:22
  • What I can see from your notes is that it only fails when running on wget `1.19.4` but it succeds in `1.17.1`. You no longer have VPS2, so then your current config is: LAPTOP, VPS1 and VPS3. Is that correct? – alariva Aug 08 '18 at 23:34
  • With the firefox extension cli2 I can see that if the url is downloaded from firefox browser the final url changes with a token added at the end of the url. If I copy all the wget text of cli2 firefox extension then that downloads on all the VPS, the problem is that every file needs a different token generation... so the question remains: why some VPS don't get a token generated for the ur: – Juan Martin Aug 08 '18 at 23:36
  • I believe this is a security-related behavior attached to wget version. This means that some implementations will follow an http redirect (302) with the generated token, and some implementations (*for some reason we are researching*) will not follow that redirect. – alariva Aug 08 '18 at 23:38
  • Can you upload the entire config files of `/etc/wgetrc` into individual pastebins for each environment? A settings comparison might bring a clue. – alariva Aug 09 '18 at 00:51
  • Sorry alariva I'm driving you crazy with wrong reports!!!! I have laptop VPS2, VPS. All of them have wget 1.17.1 but in one of the VPS's it doesn't download. The content of /etc/wgetrc is the same in laptop,vps and vps2 (I compared the files with meld): https://pastebin.com/qV9MuvkM – Juan Martin Aug 09 '18 at 06:04
  • As long as we are correctly gathering the evidence of environments, just operating systems is left as difference. Still, I honestly don't see the context is clear enough as to narrow down the problem. I would do the trials myself if that is possible. – alariva Aug 09 '18 at 06:07
  • Could it be because some VPS handle redirections in different ways? From what I see the problem is either following the redirection, with the authentication or something to do to get the token generated.... But it's the same OS, same wget version, same wgetrc file... just different machines... – Juan Martin Aug 09 '18 at 06:17
  • Another possible influece factor might be the region from which you are generating the requests. The remote server might be rejecting some regions/ip/hosts and thus not providing an authentication token. I don't believe thats the case but given the scenario as described so far, it's totally plausible. – alariva Aug 09 '18 at 06:19
  • Can you post a fresh and clear set of the three wget runs in debug mode, each telling which environment/machine (`uname -a`) is and what result you got? – alariva Aug 09 '18 at 06:22
  • The only significant difference appart from wget version I see is the REFERER in header. Usually the referer takes an important role in these types of validations, however it's now working the opposite as expected: no referer = download ok, referer = no download. I think it should be the other way around, but if the referer is being incorrectly filled then looks like normal to get rejection. https://snag.gy/fuArd8.jpg – alariva Aug 09 '18 at 06:29

0 Answers0