python http 请求 demo

其他技术

2017-06-25

43

0

技术:python3.5 + requests

运行环境:python3.5 + windows 10

demo功能:提供一个python requests 模块(http网络请求)使用demo。 抓取百度美女

安装requests模块:github: https://github.com/kennethreitz/requests

安装requests 模块
1. download the source code
2. 进入requests目录, 执行 python setup.py install, 直到运行成功

源代码

# -*- coding: utf-8 -*-
import sys, json, os, uuid, time
import requests

def downloadImageFile(imgUrl):##根据图片地址, 下载图片到本地
    local_filename = imgUrl.split('/')[-1]
    print("Download Image File=", local_filename)
    #extension = os.path.splitext(json_string['newslist'][0]['picUrl'])[1]
    r = requests.get(imgUrl, stream=True) # here we need to set stream = True parameter  
    with open('D:\\images\\'+local_filename, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)
                f.flush()
        f.close()  
    return local_filename

count=905
counter=0
while 1:
    if(counter>=600):
        time.sleep(1)
    url='http://image.baidu.com/search/avatarjson?tn=resultjsonavatarnew&ie=utf-8&word=%E7%BE%8E%E5%A5%B3&cg=girl&pn='+str(30*count)+'&rn=30&itg=0&z=0&fr=&width=&height=&lm=-1&ic=0&s=0&st=-1&gsm=960000000096'
    count=count+1;
    req = requests.get(url)
    json_string = json.loads(req.text)
    for i in range(1, len(json_string['imgs'])):
        try:
        
            downloadImageFile(json_string['imgs'][i-1]['objURL'])
        except Exception as e:
            print(e)

        counter=counter+1
        print(str(counter) +'at '+ str(time.time()))
        time.sleep(0.001)
    

 结果

结果自己看吧, 我已经下载了好多, 只是百度有下载限制

 
 

欢迎添加微信,互相学习↑↑↑ -_-

发表评论

全部评论:0条

白老虎

programming is not only to solve problems, ways to think