使用python将网页保存为文件

我是 python 新手，我的工作是尝试导出一些历史数据。我想要做的是将数百个 url 链接保存为单独的 pdf，这样我们就不必逐个单击并保存。网址是我想下载的表格的直接链接。该网页还具有用户名密码认证。我似乎无法让 python 以任何格式导出 url 链接；起初，由于用户名/密码，网页似乎不允许我访问，但在我添加 requests.get 和 auth 片段后，脚本似乎运行但没有创建导出。

正如其中一位评论者建议的 pywebcopy，我尝试了它，这个工具成功地在目标中创建了一个文件夹和一个 html 文件，并且具有正确的 url 文件名，但文件本身是空白的。我添加了身份验证部分，但没有任何区别，因为保存的 html 文件仍然是空白的。

import requests

requests.get('main website url', auth=('username','password'))

urls = ['url1','url2','url3' etc]

output_dir = 'folder on my drive'

for url in urls:
    response = requests.get(url)
    if response.status_code == 200:
        file_path = os.path.join(output_dir, os.path.basename(url))
        with open(file_path, 'wb') as f:
            f.write(response.content)

这是http响应

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): apps.bell.com:443
send: b'GET / HTTP/1.1\r\nHost: apps.bell.com\r\nUser-Agent: python-requests/2.27.1\r\nAccept-Encoding: gzip, deflate, br\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 302 \r\n'
header: Set-Cookie: JSESSIONID=B521B7D0D66595F87A12E174CA2C4CAD; Path=/; Secure; HttpOnly
header: Location: https://apps.bell.com/apps/bell/bell.bellmain
header: Content-Type: text/html;charset=ISO-8859-1
header: Content-Length: 0
header: Date: Mon, 16 May 2022 00:26:34 GMT
header: Keep-Alive: timeout=1
header: Connection: keep-alive
DEBUG:urllib3.connectionpool:https://apps.bell.com:443 "GET / HTTP/1.1" 302 0
send: b'GET /apps/bell/bell.bellmain HTTP/1.1\r\nHost: apps.bell.com\r\nUser-Agent: python-requests/2.27.1\r\nAccept-Encoding: gzip, deflate, br\r\nAccept: */*\r\nConnection: keep-alive\r\nCookie: JSESSIONID=B521B7D0D66595F87A12E174CA2C4CAD\r\n\r\n'
reply: 'HTTP/1.1 401 \r\n'
header: WWW-Authenticate: Basic realm="bellProduction System - V8MU.Q3"
header: Content-Type: text/html
header: Content-Length: 522
header: Date: Mon, 16 May 2022 00:26:34 GMT
header: Keep-Alive: timeout=1
header: Connection: keep-alive
DEBUG:urllib3.connectionpool:https://apps.bell.com:443 "GET /apps/bell/bell.bellmain HTTP/1.1" 401 522
PS >