About w3af_api

发布时间:November 23, 2016 // 分类:运维工作,工作日志,开发笔记,代码学习,linux,python,生活琐事 // No Comments

今天看到了一个saas产品,和作者聊了下,发现是基于w3af_api来实现的。然后自己补充了其他的类型.感觉很厉害的样子。于是跑过来看了下w3af。相关的文档在这里

w3af算的上是老牌的东西了。反正我是比较少用的,总是感觉效果没有理想的那么好。比如它的爬虫模块太久没有更新了.导致现在出现的很多动态脚本的结果没发准确抓取到。对比下

- http://testphp.acunetix.com/
- http://testphp.acunetix.com/AJAX/
- http://testphp.acunetix.com/AJAX/index.php
- http://testphp.acunetix.com/AJAX/styles.css
- http://testphp.acunetix.com/Flash/
- http://testphp.acunetix.com/Flash/add.fla
- http://testphp.acunetix.com/Flash/add.swf
- http://testphp.acunetix.com/Mod_Rewrite_Shop/
- http://testphp.acunetix.com/Mod_Rewrite_Shop/images/1.jpg
- http://testphp.acunetix.com/Mod_Rewrite_Shop/images/2.jpg
- http://testphp.acunetix.com/Mod_Rewrite_Shop/images/3.jpg
- http://testphp.acunetix.com/artists.php
- http://testphp.acunetix.com/cart.php
- http://testphp.acunetix.com/categories.php
- http://testphp.acunetix.com/disclaimer.php
- http://testphp.acunetix.com/guestbook.php
- http://testphp.acunetix.com/hpp/
- http://testphp.acunetix.com/hpp/params.php
- http://testphp.acunetix.com/images/logo.gif
- http://testphp.acunetix.com/images/remark.gif
- http://testphp.acunetix.com/index.php
- http://testphp.acunetix.com/listproducts.php
- http://testphp.acunetix.com/login.php
- http://testphp.acunetix.com/product.php
- http://testphp.acunetix.com/redir.php
- http://testphp.acunetix.com/search.php
- http://testphp.acunetix.com/secured/
- http://testphp.acunetix.com/secured/newuser.php
- http://testphp.acunetix.com/secured/style.css
- http://testphp.acunetix.com/showimage.php
- http://testphp.acunetix.com/signup.php
- http://testphp.acunetix.com/style.css
- http://testphp.acunetix.com/userinfo.php

这个是它抓取到的。然后自己前几天琢磨的crawl抓到的【在抓取前先fuzz了dir。所以基本满足需求】

主题不在这边。主要是针对w3af_api。关于它的文档可以看这边。简答的描述下

1.启动,主要是两个方式,一个是直接运行

$ ./w3af_api
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

另外一个是docker

$ cd extras/docker/scripts/
$ ./w3af_api_docker
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

2.认证。可以自行更换密码的。密码默认的加密方式是sha512sum。

生成密码

$ echo -n "secret" | sha512sum
bd2b1aaf7ef4f09be9f52ce2d8d599674d81aa9d6a4421696dc4d93dd0619d682ce56b4d64a9ef097761ced99e0f67265b5f76085e5b0ee7ca4696b2ad6fe2b2  -

$ ./w3af_api -p "bd2b1aaf7ef4f09be9f52ce2d8d599674d81aa9d6a4421696dc4d93dd0619d682ce56b4d64a9ef097761ced99e0f67265b5f76085e5b0ee7ca4696b2ad6fe2b2"
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

也可以把账户密码等信息写入yml配置文件来加载启动。

3.api使用方式

开始一个新的扫描 [POST] /scans/
查看扫描状态 GET /scans/0/status
获取相关的漏洞信息使用 GET /scan/kb/
删除相关的信息  DELETE /scans/0/
获取扫描信息  GET  /scans/
暂停扫描  GET /scans/0/pause
停止扫描  GET /scans/0/stop
查看扫描日志  GET /scans/0/log

实际栗子来尝试一次扫描

import requests
import json

data = {'scan_profile': file('../core/w3af/profiles/full_audit.pw3af').read(),
        'target_urls': ['http://testphp.acunetix.com']}

response = requests.post('http://127.0.0.1:5000/scans/',
                         data=json.dumps(data),
                         headers={'content-type': 'application/json'})
                         
print response.text
scan_profile    必须包含的内容w3af扫描配置文件(文件名)
target_urls      w3af要进行爬虫的url列表

查看扫描状态

查看扫描状态

查看相关的漏洞信息

具体某个漏洞的信息

{
  "attributes": {
    "db": "MySQL database",
    "error": "mysql_"
  },
  "cwe_ids": [
    "89"
  ],
  "cwe_urls": [
    "https://cwe.mitre.org/data/definitions/89.html"
  ],
  "desc": "SQL injection in a MySQL database was found at: \"http://testphp.acunetix.com/userinfo.php\", using HTTP method POST. The sent post-data was: \"uname=a%27b%22c%27d%22&pass=FrAmE30.\" which modifies the \"uname\" parameter.",
  "fix_effort": 50,
  "fix_guidance": "The only proven method to prevent against SQL injection attacks while still maintaining full application functionality is to use parameterized queries (also known as prepared statements). When utilising this method of querying the database, any value supplied by the client will be handled as a string value rather than part of the SQL query.\n\nAdditionally, when utilising parameterized queries, the database engine will automatically check to make sure the string being used matches that of the column. For example, the database engine will check that the user supplied input is an integer if the database column is configured to contain integers.",
  "highlight": [
    "mysql_"
  ],
  "href": "/scans/0/kb/29",
  "id": 29,
  "long_description": "Due to the requirement for dynamic content of today's web applications, many rely on a database backend to store data that will be called upon and processed by the web application (or other programs). Web applications retrieve data from the database by using Structured Query Language (SQL) queries.\n\nTo meet demands of many developers, database servers (such as MSSQL, MySQL, Oracle etc.) have additional built-in functionality that can allow extensive control of the database and interaction with the host operating system itself. An SQL injection occurs when a value originating from the client's request is used within a SQL query without prior sanitisation. This could allow cyber-criminals to execute arbitrary SQL code and steal data or use the additional functionality of the database server to take control of more server components.\n\nThe successful exploitation of a SQL injection can be devastating to an organisation and is one of the most commonly exploited web application vulnerabilities.\n\nThis injection was detected as the tool was able to cause the server to respond to the request with a database related error.",
  "name": "SQL injection",
  "owasp_top_10_references": [
    {
      "link": "https://www.owasp.org/index.php/Top_10_2013-A1",
      "owasp_version": "2013",
      "risk_id": 1
    }
  ],
  "plugin_name": "sqli",
  "references": [
    {
      "title": "SecuriTeam",
      "url": "http://www.securiteam.com/securityreviews/5DP0N1P76E.html"
    },
    {
      "title": "Wikipedia",
      "url": "http://en.wikipedia.org/wiki/SQL_injection"
    },
    {
      "title": "OWASP",
      "url": "https://www.owasp.org/index.php/SQL_Injection"
    },
    {
      "title": "WASC",
      "url": "http://projects.webappsec.org/w/page/13246963/SQL%20Injection"
    },
    {
      "title": "W3 Schools",
      "url": "http://www.w3schools.com/sql/sql_injection.asp"
    },
    {
      "title": "UnixWiz",
      "url": "http://unixwiz.net/techtips/sql-injection.html"
    }
  ],
  "response_ids": [
    1494
  ],
  "severity": "High",
  "tags": [
    "web",
    "sql",
    "injection",
    "database",
    "error"
  ],
  "traffic_hrefs": [
    "/scans/0/traffic/1494"
  ],
  "uniq_id": "82f91e8c-759b-43b9-82cb-59ff9a38a836",
  "url": "http://testphp.acunetix.com/userinfo.php",
  "var": "uname",
  "vulndb_id": 45,
  "wasc_ids": [],
  "wasc_urls": []
}

感觉有这些差不多了。可以开始扫描,暂停,停止,删除。还能获取到具体的某个漏洞细节以及修复方案。加上api独有的特效,是可以做分布式的.

import pika
import requests
import json
import sys
import time
import sqlalchemy as db
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
import os

# database stuffs
Base = declarative_base()

# scan
class Scan(Base):
    __tablename__ = 'scans'
    id = db.Column(db.Integer, primary_key = True)
    relative_id = db.Column(db.Integer)
    description = db.Column(db.Text)
    target_url = db.Column(db.String(128))
    start_time = db.Column(db.Time)
    scan_time = db.Column(db.Time, nullable=True)
    profile = db.Column(db.String(32))
    status = db.Column(db.String(32))
    deleted = db.Column(db.Boolean, default=False)
    run_instance = db.Column(db.Unicode(128))
    num_vulns = db.Column(db.Integer)
    vulns = db.orm.relationship("Vulnerability", back_populates="scan")
    user_id = db.Column(db.String(40))

    def __repr__(self):
        return '<Scan %d>' % self.id

# vuln
class Vulnerability(Base):
    __tablename__ = 'vulns'
    id = db.Column(db.Integer, primary_key = True)
    relative_id = db.Column(db.Integer) # relative to scans
    stored_json = db.Column(db.Text) # inefficient, might fix later
    deleted = db.Column(db.Boolean, default=False)
    false_positive = db.Column(db.Boolean, default=False)
    scan_id = db.Column(db.Integer, db.ForeignKey('scans.id'))
    scan = db.orm.relationship("Scan", back_populates="vulns")

    def __init__(self, id, json, scan_id):
        self.relative_id = id
        self.stored_json = json
        self.scan_id = scan_id

    def __repr__(self):
        return '<Vuln %d>' % self.id

engine = db.create_engine(os.environ.get('SQLALCHEMY_CONN_STRING'))
Session = sessionmaker(bind=engine)
sess = Session()

credentials = pika.PlainCredentials(os.environ.get('TASKQUEUE_USER'), os.environ.get('TASKQUEUE_PASS'))
con = pika.BlockingConnection(pika.ConnectionParameters(host=os.environ.get('TASKQUEUE_HOST'),credentials=credentials))

channelTask = con.channel()
channelTask.queue_declare(queue='task', durable=True)

channelResult = con.channel()
channelResult.queue_declare(queue='result')

# URL to w3af REST API interface instance
server = sys.argv[1]

vul_cnt = 0

def freeServer(sv, href):
    r = requests.delete(sv + href)
    print r.text

def isFree(sv):
    r = requests.get(sv + '/scans/')
    print r.text
    items = json.loads(r.text)['items']
    if len(items) == 0:
        return True
    # number of items > 0
    item = items[0]
    if item['status'] == 'Stopped':
        freeServer(sv, item['href'])
        return True
    return False

def sendTaskDone(server, href):
    data = {}
    data['server'] = server
    data['href'] = href
    message = json.dumps(data)
    channelResult.basic_publish(exchange='',
                        routing_key='result',
                        body=message)

def scann(target):
    data = {'scan_profile': file('../core/w3af/profiles/full_audit.pw3af').read(),
        'target_urls': [target]}
    response = requests.post(server + '/scans/',
                        data=json.dumps(data),
                        headers={'content-type': 'application/json'})

    print response.status_code
    print response.data
    print response.headers

def getVul(sv, href):
    r = requests.get(sv + href)
    #db.insert(r.text)

def getVulsList(sv, href):
    global vul_cnt
    r = requests.get(sv + href + 'kb')
    vuls = json.loads(r.text)['items']
    l = len(vuls)
    if l > vuls_cnt:
        for vul in vuls:
            if vul['id'] >= vul_cnt:
                getVul(sv, vul['href'])
    vul_cnt = l
        
# on receiving message
def callback(ch, method, properties, body):
    print('Get message %s', body)
    task = json.loads(body)
    scann(task['target'])
    task_done = False
    time.sleep(1)
    step = 0
    last_vuln_len = 0
    sv = server
    scan = sess.query(Scan).filter_by(id=task['scan_id']).first()
    # tell gateway server that the task is loaded on this instance
    scan.run_instance = server
    while True:
        # update scan status; check if freed
        list_scans = json.loads(requests.get(sv + '/scans/').text)['items'] # currently just 1
        if (len(list_scans) == 0): # freed
            break
        currentpath = list_scans[0]['href']
        # update vuln list
        r = requests.get(sv + currentpath + '/kb/')
        items = json.loads(r.text)['items'] 
        for i in xrange(last_vuln_len, len(items)):
            v = Vulnerability(i+1, requests.get(sv + items[i]['href']).text, task['scan_id'])
            sess.add(v)
            sess.commit()
            scan.num_vulns += 1
        last_vuln_len = len(items)
        scan.status = list_scans[0]['status']
        sess.commit()
        if scan.status == 'Stopped' and not task_done:
            task_done = True
            requests.delete(sv + currentpath)
        step += 1
        if step == 9:
            con.process_data_events() # MQ heartbeat
            step = 0
        time.sleep(5) # avoid over consumption
    # TODO: send mails to list when the scan is stopped or completed
    print 'DOne'
    ch.basic_ack(delivery_tag=method.delivery_tag)
#print getServerStatus(server)


channelTask.basic_qos(prefetch_count=1)
channelTask.basic_consume(callback, queue='task')

print '[*] Waiting for message'

channelTask.start_consuming()

 

在线cms识别以及bugscan插件调用

发布时间:October 23, 2016 // 分类:开发笔记,linux,python,windows // No Comments

想着自己搞的话估计是比较符合自己的需求。但是问题就是太耗时了,估计覆盖面也不广泛。

恰逢遇到了http://whatweb.bugscaner.com/这个网站,发现它的覆盖面还是不错的。常见的cms都整合过去了。测试了几个发现误报率还是在可以接受的范围内.于是自动化。一个简单的demo.缺点是只能访问http一类的.https的不支持。它提交的时候会自动去掉http|https://

#!/usr/bin/python
import re
import json
import requests



def whatcms(url):
    headers = {"Content-Type":"application/x-www-form-urlencoded; charset=UTF-8",
        "Referer":"http://whatweb.bugscaner.com/look/",
        }
    """
    try:
        res = requests.get('http://whatweb.bugscaner.com/look/',timeout=60, verify=False)
        if res.status_code==200:
            hashes = re.findall(r'value="(.*?)" name="hash" id="hash"',res.content)[0]
    except Exception as e:
        print str(e)
        return False
    """
    data = "url=%s&hash=0eca8914342fc63f5a2ef5246b7a3b14_7289fd8cf7f420f594ac165e475f1479"%(url)
    try:
        respone = requests.post("http://whatweb.bugscaner.com/what/",data=data,headers=headers,timeout=60, verify=False)
        if int(respone.status_code)==200:
            result = json.loads(respone.content)
            if len(result["cms"])>0:
                return result["cms"]
            else:
                return "www"
    except Exception as e:
        print str(e)
        return "www"
        
if __name__ == '__main__':
    import sys
    url = sys.avg[1]
    print whatcms(url)

无法识别的自动判断为www。既然都可以完美搞定了。接下来开始整合插件.我的想法是先分类.读取文件内容中的service。然后再把文件名称和servvice存进数据库.方便以后调用。简单的来个小脚本

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re,os,glob
from mysql_class import MySQL
"""
logging.basicConfig(
    level=logging.DEBUG,
    format="[%(asctime)s] %(levelname)s: %(message)s")
"""
"""
1.识别具体的cms
2.从数据库获取cms--如果没有获取到考虑全部便利
3.输出结果
"""
def timestamp():
    return str(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))

"""
DROP TABLE IF EXISTS `bugscan`;
CREATE TABLE `bugscan` (
    `id` int(11) NOT NULL AUTO_INCREMENT,
    `service` varchar(256) COLLATE utf8_bin DEFAULT NULL,
    `filename` varchar(256) COLLATE utf8_bin DEFAULT NULL,
    `time` varchar(256) COLLATE utf8_bin DEFAULT NULL,
    PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
SET FOREIGN_KEY_CHECKS = 1;
"""
dbconfig = {'host':'127.0.0.1','port': 3306,'user':'root','passwd':'root123','db':'proscan','charset':'utf8'}
db = MySQL(dbconfig)

def insert(filename):
    file_data = open(filename,'rb').read()
    service = re.findall(r"if service.*==(.*?):",file_data)
    if len(service)>0:
        servi = service[0].replace("'", "").replace("\"", "").replace(" ", "")
        
        sqlInsert = "insert into `bugscan`(id,service,filename,time) values ('','%s','%s','%s');" % (str(servi),str(filename.replace('./bugscannew/','')),str(timestamp()))
        print sqlInsert
        #db.query(sql=sqlInsert)

for filename in glob.glob(r'./bugscannew/*.py'):
    insert(filename)

然后思考下怎么调用这个具体的插件来进行判断。其实想了好久。直到前不久空着有空看了pocscan。发现这个方式不错.把文件加入到pypath中.然后from xxx import audit 然后就完美解决这个问题了.

def import_poc(pyfile,url):
    poc_path = os.getcwd()+"/bugscannew/"
    path = poc_path + pyfile + ".py"
    filename = path.split("/")[-1].split(".py")[0]
    sys.path.append(poc_path)
    poc0 = imp.load_source('audit', path)
    audit_function = poc0.audit
    from dummy import *
    audit_function.func_globals.update(locals())
    ret = audit_function(url)
    if ret is not None and 'None' not in ret:
        #print ret
        return ret

暂时没完美调用的方式.简单的贴个demo

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re,os
import imp,sys
import time,json
import logging,glob
import requests
from mysql_class import MySQL
"""
logging.basicConfig(
    level=logging.DEBUG,
    format="[%(asctime)s] %(levelname)s: %(message)s")
"""
"""
1.识别具体的cms
2.从数据库获取cms--如果没有获取到考虑全部便利
3.输出结果
"""
def timestamp():
    return str(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()))

"""
DROP TABLE IF EXISTS `bugscan`;
CREATE TABLE `bugscan` (
    `id` int(11) NOT NULL AUTO_INCREMENT,
    `service` varchar(256) COLLATE utf8_bin DEFAULT NULL,
    `filename` varchar(256) COLLATE utf8_bin DEFAULT NULL,
    `time` varchar(256) COLLATE utf8_bin DEFAULT NULL,
    PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
SET FOREIGN_KEY_CHECKS = 1;
"""
dbconfig = {'host':'127.0.0.1','port': 3306,'user':'root','passwd':'root123','db':'proscan','charset':'utf8'}
db = MySQL(dbconfig)

def insert(filename):
    file_data = open(filename,'rb').read()
    service = re.findall(r"if service.*==(.*?):",file_data)
    if len(service)>0:
        servi = service[0].replace("'", "").replace("\"", "").replace(" ", "")
        
        sqlInsert = "insert into `bugscan`(id,service,filename,time) values ('','%s','%s','%s');" % (str(servi),str(filename.replace('./bugscannew/','')),str(timestamp()))
        print sqlInsert
        #db.query(sql=sqlInsert)
        #print servi,filename

def check(service,url):
    if service == 'www':
        sqlsearch = "select  filename from  `bugscan` where service = '%s'" %(service)
    elif service != 'www':
        sqlsearch = "select  filename from  `bugscan` where service = 'www' or service = '%s'" %(service)
    print sqlsearch 
    if int(db.query(sql=sqlsearch))>0:
        result = db.fetchAllRows()
        for row in result:
            #return result
            for colum in row:
                colum = colum.replace(".py","")
                import_poc(colum,url)

def import_poc(pyfile,url):
    poc_path = os.getcwd()+"/bugscannew/"
    path = poc_path + pyfile + ".py"
    filename = path.split("/")[-1].split(".py")[0]
    sys.path.append(poc_path)
    poc0 = imp.load_source('audit', path)
    audit_function = poc0.audit
    from dummy import *
    audit_function.func_globals.update(locals())
    ret = audit_function(url)
    if ret is not None and 'None' not in ret:
        #print ret
        return ret

def whatcms(url):
    headers = {"Content-Type":"application/x-www-form-urlencoded; charset=UTF-8",
        "Referer":"http://whatweb.bugscaner.com/look/",
        }
    """
    try:
        res = requests.get('http://whatweb.bugscaner.com/look/',timeout=60, verify=False)
        if res.status_code==200:
            hashes = re.findall(r'value="(.*?)" name="hash" id="hash"',res.content)[0]
    except Exception as e:
        print str(e)
        return False
    """
    data = "url=%s&hash=0eca8914342fc63f5a2ef5246b7a3b14_7289fd8cf7f420f594ac165e475f1479"%(url)
    try:
        respone = requests.post("http://whatweb.bugscaner.com/what/",data=data,headers=headers,timeout=60, verify=False)
        if int(respone.status_code)==200:
            result = json.loads(respone.content)
            if len(result["cms"])>0:
                return result["cms"]
            else:
                return "www"
    except Exception as e:
        print str(e)
        return "www"
        
if __name__ == '__main__':
    #for filename in glob.glob(r'./bugscannew/*.py'):
    #   insert(filename)
    url = "http://0day5.com/"
    print check(whatcms(url),url)

其实还有纰漏。比如在调用那块可以考虑下采用多线程来加快速度.还有就是可能出现如果cms无法识别出来。结果肯定不准确。如果全部load进来fuzz一次太耗时了。

得到琦神的demo。貌似更暴力,全加载fuzz一次

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# papapa.py
import re
import socket
import sys
import os
import urlparse
import time
from dummy.common import *
import util
from dummy import *
import importlib
import threading
import Queue as que


class Worker(threading.Thread):  # 处理工作请求
    def __init__(self, workQueue, resultQueue, **kwds):
        threading.Thread.__init__(self, **kwds)
        self.setDaemon(True)
        self.workQueue = workQueue
        self.resultQueue = resultQueue

    def run(self):
        while 1:
            try:
                callable, args, kwds = self.workQueue.get(False)  # get task
                res = callable(*args, **kwds)
                self.resultQueue.put(res)  # put result
            except que.Empty:
                break


class WorkManager:  # 线程池管理,创建
    def __init__(self, num_of_workers=10):
        self.workQueue = que.Queue()  # 请求队列
        self.resultQueue = que.Queue()  # 输出结果的队列
        self.workers = []
        self._recruitThreads(num_of_workers)

    def _recruitThreads(self, num_of_workers):
        for i in range(num_of_workers):
            worker = Worker(self.workQueue, self.resultQueue)  # 创建工作线程
            self.workers.append(worker)  # 加入到线程队列

    def start(self):
        for w in self.workers:
            w.start()

    def wait_for_complete(self):
        while len(self.workers):
            worker = self.workers.pop()  # 从池中取出一个线程处理请求
            worker.join()
            if worker.isAlive() and not self.workQueue.empty():
                self.workers.append(worker)  # 重新加入线程池中
        #logging.info('All jobs were complete.')

    def add_job(self, callable, *args, **kwds):
        self.workQueue.put((callable, args, kwds))  # 向工作队列中加入请求

    def get_result(self, *args, **kwds):
        return self.resultQueue.get(*args, **kwds)
"""
lst=os.listdir(os.getcwd())
pocList =(','.join(c.strip('.py') for c in lst if os.path.isfile(c) and c.endswith('.py'))).split(',')
for line in pocList:
    try:
        #print line
        xxoo = importlib.import_module(line)
        xxoo.curl = miniCurl.Curl()
        xxoo.security_hole = security_hole
        xxoo.task_push = task_push
        xxoo.util =util
        xxoo.security_warning = security_warning
        xxoo.security_note = security_note
        xxoo.security_info = security_info
        xxoo.time = time
        xxoo.audit('http://0day5.com')
    except Exception as e:
        print line,e
"""

def bugscan(line,url):
    #print line,url
    try:
        xxoo = importlib.import_module(line)
        xxoo.curl = miniCurl.Curl()
        xxoo.security_hole = security_hole
        xxoo.task_push = task_push
        xxoo.util =util
        xxoo.security_warning = security_warning
        xxoo.security_note = security_note
        xxoo.security_info = security_info
        xxoo.time = time
        xxoo.audit(url)
    except Exception as e:
        #print line,e
        pass

def main(url):
    wm = WorkManager(20)
    lst=os.listdir(os.getcwd())
    pocList =(','.join(c.strip('.py') for c in lst if os.path.isfile(c) and c.endswith('.py'))).split(',')
    for line in pocList:
        if 'apa' not in line:
    wm.start()
    wm.wait_for_complete()
start = time.time()
main('http://0day5.com/')
print time.time()-start

准确率堪忧啊,仅供参考

使用python-nmap不出https踩到的坑

发布时间:September 1, 2016 // 分类:开发笔记,工作日志,运维工作,linux,python,windows,生活琐事 // No Comments

今天在使用python-nmap来进行扫描入库的时候发现https端口硬生生的给判断成了http

仔细测试了好几次都是这样子。后来果断的不服气,仔细看了下扫描的参数。发现python-nmap调用的时候会强制加上-ox -这个参数

正常扫描是

nmap 45.33.49.119 -p T:443 -Pn -sV --script=banner

然而经过python-nmap以后就是

nmap -oX - 45.33.49.119 -p T:443 -Pn -sV --script=banner
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE nmaprun>
<?xml-stylesheet href="file:///usr/local/bin/../share/nmap/nmap.xsl" type="text/xsl"?>
<!-- Nmap 7.12 scan initiated Thu Sep  1 00:02:07 2016 as: nmap -oX - -p T:443 -Pn -sV -&#45;script=banner 45.33.49.119 -->
<nmaprun scanner="nmap" args="nmap -oX - -p T:443 -Pn -sV -&#45;script=banner 45.33.49.119" start="1472659327" startstr="Thu Sep  1 00:02:07 2016" version="7.12" xmloutputversion="1.04">
<scaninfo type="connect" protocol="tcp" numservices="1" services="443"/>
<verbose level="0"/>
<debugging level="0"/>
<host starttime="1472659328" endtime="1472659364"><status state="up" reason="user-set" reason_ttl="0"/>
<address addr="45.33.49.119" addrtype="ipv4"/>
<hostnames>
<hostname name="ack.nmap.org" type="PTR"/>
</hostnames>
<ports><port protocol="tcp" portid="443"><state state="open" reason="syn-ack" reason_ttl="0"/><service name="http" product="Apache httpd" version="2.4.6" extrainfo="(CentOS)" tunnel="ssl" method="probed" conf="10"><cpe>cpe:/a:apache:http_server:2.4.6</cpe></service><script id="http-server-header" output="Apache/2.4.6 (CentOS)"><elem>Apache/2.4.6 (CentOS)</elem>
</script></port>
</ports>
<times srtt="191238" rttvar="191238" to="956190"/>
</host>
<runstats><finished time="1472659364" timestr="Thu Sep  1 00:02:44 2016" elapsed="36.65" summary="Nmap done at Thu Sep  1 00:02:44 2016; 1 IP address (1 host up) scanned in 36.65 seconds" exit="success"/><hosts up="1" down="0" total="1"/>
</runstats>
</nmaprun>

经过格式化以后看到的内容是

其中的一个参数tunnel.但是看了下https://bitbucket.org/xael/python-nmap/raw/8ed37a2ac20d6ef26ead60d36f739f4679fcdc3e/nmap/nmap.py这里的内容。发现没有与之关联的。

for dport in dhost.findall('ports/port'):
                # protocol
                proto = dport.get('protocol')
                # port number converted as integer
                port =  int(dport.get('portid'))
                # state of the port
                state = dport.find('state').get('state')
                # reason
                reason = dport.find('state').get('reason')
                # name, product, version, extra info and conf if any
                name = product = version = extrainfo = conf = cpe = ''
                for dname in dport.findall('service'):
                    name = dname.get('name')
                    if dname.get('product'):
                        product = dname.get('product')
                    if dname.get('version'):
                        version = dname.get('version')
                    if dname.get('extrainfo'):
                        extrainfo = dname.get('extrainfo')
                    if dname.get('conf'):
                        conf = dname.get('conf')

                    for dcpe in dname.findall('cpe'):
                        cpe = dcpe.text
                # store everything
                if not proto in list(scan_result['scan'][host].keys()):
                    scan_result['scan'][host][proto] = {}

                scan_result['scan'][host][proto][port] = {'state': state,
                                                          'reason': reason,
                                                          'name': name,
                                                          'product': product,
                                                          'version': version,
                                                          'extrainfo': extrainfo,
                                                          'conf': conf,
                                                          'cpe': cpe}

试想下如果把name以及tunnel取出来同时匹配不就好了。于是对此进行修改。410-440行

                name = product = version = extrainfo = conf = cpe = tunnel =''
                for dname in dport.findall('service'):
                    name = dname.get('name')
                    if dname.get('product'):
                        product = dname.get('product')
                    if dname.get('version'):
                        version = dname.get('version')
                    if dname.get('extrainfo'):
                        extrainfo = dname.get('extrainfo')
                    if dname.get('conf'):
                        conf = dname.get('conf')
                    if dname.get('tunnel'):
                        tunnel = dname.get('tunnel')

                    for dcpe in dname.findall('cpe'):
                        cpe = dcpe.text
                # store everything
                if not proto in list(scan_result['scan'][host].keys()):
                    scan_result['scan'][host][proto] = {}

                scan_result['scan'][host][proto][port] = {'state': state,
                                                          'reason': reason,
                                                          'name': name,
                                                          'product': product,
                                                          'version': version,
                                                          'extrainfo': extrainfo,
                                                          'conf': conf,
                                                          'tunnel':tunnel,
                                                          'cpe': cpe}

还有在654-670行里面增加我们添加的tunnel

        csv_ouput = csv.writer(fd, delimiter=';')
        csv_header = [
            'host',
            'hostname',
            'hostname_type',
            'protocol',
            'port',
            'name',
            'state',
            'product',
            'extrainfo',
            'reason',
            'version',
            'conf',
            'tunnel',
            'cpe'
            ]

然后我们import这个文件。在获取的内容里面进行判断.如果namehttp的同时tunnelssl,则判断为https

        for targetHost in scanner.all_hosts():
            if scanner[targetHost].state() == 'up' and scanner[targetHost]['tcp']:
                for targetport in scanner[targetHost]['tcp']:
                    #print(scanner[targetHost]['tcp'][int(targetport)])
                    if scanner[targetHost]['tcp'][int(targetport)]['state'] == 'open' and scanner[targetHost]['tcp'][int(targetport)]['product']!='tcpwrapped':
                        if scanner[targetHost]['tcp'][int(targetport)]['name']=='http' and scanner[targetHost]['tcp'][int(targetport)]['tunnel'] == 'ssl':
                            scanner[targetHost]['tcp'][int(targetport)]['name'] = 'https'
                        else:
                            scanner[targetHost]['tcp'][int(targetport)]['name'] = scanner[targetHost]['tcp'][int(targetport)]['name']
                        print(domain+'\t'+targetHosts+'\t'+str(targetport) + '\t' + scanner[targetHost]['tcp'][int(targetport)]['name'] + '\t' + scanner[targetHost]['tcp'][int(targetport)]['product']+scanner[targetHost]['tcp'][int(targetport)]['version'])
                        #if scanner[targetHost]['tcp'][int(targetport)]['name'] in ["https","http"]:

改造后的文件扫描结果

python 旁站查询

发布时间:July 30, 2016 // 分类:开发笔记,工作日志,代码学习,linux,python,windows // No Comments

旁站查询来源:

效果图如下

#!/usr/bin/env python
#encoding: utf-8
import re
import sys
import json
import time
import requests
import urllib
import requests.packages.urllib3
from multiprocessing import Pool
from BeautifulSoup import BeautifulSoup
requests.packages.urllib3.disable_warnings()

headers = {'User-Agent' : 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20'}

def links_ip(host):   
    '''
    查询同IP网站
    '''
    ip2hosts = []
    ip2hosts.append("http://"+host)
    try:
        source = requests.get('http://i.links.cn/sameip/' + host + '.html', headers=headers,verify=False)
        soup = BeautifulSoup(source.text)
        divs = soup.findAll(style="word-break:break-all")
        
        if divs == []: #抓取结果为空
            print 'Sorry! Not found!'
            return ip2hosts 
        for div in divs:
            #print div.a.string
            ip2hosts.append(div.a.string)
    except Exception, e:
        print str(e)
        return ip2hosts
    return ip2hosts

def ip2host_get(host):
    ip2hosts = []
    ip2hosts.append("http://"+host)
    try:
        req=requests.get('http://www.ip2hosts.com/search.php?ip='+str(host), headers=headers,verify=False)
        src=req.content
        if src.find('result') != -1:
            result = json.loads(src)['result']
            ip = json.loads(src)['ip']
            if len(result)>0:
                for item in result:
                    if len(item)>0:
                        #log(scan_type,host,port,str(item))
                        ip2hosts.append(item)
    except Exception, e:
        print str(e)
        return ip2hosts
    return ip2hosts


def filter(host):
    '''
        打不开的网站...
    '''
    try:
        response = requests.get(host, headers=headers ,verify=False)
        server = response.headers['Server']
        title = re.findall(r'<title>(.*?)</title>',response.content)[0]
    except Exception,e:
        #print "%s" % str(e)
        #print host
        pass
    else:
        print host,server

def aizhan(host):
    ip2hosts = []
    ip2hosts.append("http://"+host)
    regexp = r'''<a href="[^']+?([^']+?)/" rel="nofollow" target="_blank">\1</a>'''
    regexp_next = r'''<a href="http://dns.aizhan.com/[^/]+?/%d/">%d</a>'''
    url = 'http://dns.aizhan.com/%s/%d/'

    page = 1
    while True:
        if page > 2:
            time.sleep(1)   #防止拒绝访问
        req = requests.get(url % (host , page) ,headers=headers ,verify=False)
        try:
            html = req.content.decode('utf-8')  #取得页面
            if req.status_code == 400:
                break
        except Exception as e:
            print str(e)
            pass
        for site in re.findall(regexp , html):
            ip2hosts.append("http://"+site)
        if re.search(regexp_next % (page+1 , page+1) , html) is None:
            return ip2hosts
            break
        page += 1

    return ip2hosts

def chinaz(host):
    ip2hosts = []
    ip2hosts.append("http://"+host)
    regexp = r'''<a href='[^']+?([^']+?)' target=_blank>\1</a>'''
    regexp_next = r'''<a href="javascript:" val="%d" class="item[^"]*?">%d</a>'''
    url = 'http://s.tool.chinaz.com/same?s=%s&page=%d'

    page = 1
    while True:
        if page > 1:
            time.sleep(1)   #防止拒绝访问
        req = requests.get(url % (host , page) , headers=headers ,verify=False)
        html = req.content.decode('utf-8')  #取得页面
        for site in re.findall(regexp , html):
            ip2hosts.append("http://"+site)
        if re.search(regexp_next % (page+1 , page+1) , html) is None:
            return ip2hosts
            break
        page += 1
    return ip2hosts

def same_ip(host):
    mydomains = []
    mydomains.extend(ip2host_get(host))
    mydomains.extend(links_ip(host))
    mydomains.extend(aizhan(host))
    mydomains.extend(chinaz(host))
    mydomains = list(set(mydomains))
    p = Pool()
    for host in mydomains:
        p.apply_async(filter, args=(host,))
    p.close()
    p.join()


if __name__=="__main__":
    if len(sys.argv) == 2:
        same_ip(sys.argv[1])
    else:
        print ("usage: %s host" % sys.argv[0])
        sys.exit(-1)

 

python获取http代理

发布时间:July 24, 2016 // 分类:开发笔记,工作日志,运维工作,linux,python,windows // 7 Comments

主要是从http://www.ip181.com/ http://www.kuaidaili.com/以及http://www.66ip.com/获取相关的代理信息,并分别访问v2ex.com以及guokr.com以进行验证代理的可靠性。

# -*- coding=utf8 -*-
"""
    从网上爬取HTTPS代理
"""
import re
import sys
import time
import Queue
import logging
import requests
import threading
from pyquery import PyQuery
import requests.packages.urllib3
requests.packages.urllib3.disable_warnings()


#logging.basicConfig(
#    level=logging.DEBUG,
#    format="[%(asctime)s] %(levelname)s: %(message)s")

class Worker(threading.Thread):  # 处理工作请求
    def __init__(self, workQueue, resultQueue, **kwds):
        threading.Thread.__init__(self, **kwds)
        self.setDaemon(True)
        self.workQueue = workQueue
        self.resultQueue = resultQueue

    def run(self):
        while 1:
            try:
                callable, args, kwds = self.workQueue.get(False)  # get task
                res = callable(*args, **kwds)
                self.resultQueue.put(res)  # put result
            except Queue.Empty:
                break


class WorkManager:  # 线程池管理,创建
    def __init__(self, num_of_workers=10):
        self.workQueue = Queue.Queue()  # 请求队列
        self.resultQueue = Queue.Queue()  # 输出结果的队列
        self.workers = []
        self._recruitThreads(num_of_workers)

    def _recruitThreads(self, num_of_workers):
        for i in range(num_of_workers):
            worker = Worker(self.workQueue, self.resultQueue)  # 创建工作线程
            self.workers.append(worker)  # 加入到线程队列

    def start(self):
        for w in self.workers:
            w.start()

    def wait_for_complete(self):
        while len(self.workers):
            worker = self.workers.pop()  # 从池中取出一个线程处理请求
            worker.join()
            if worker.isAlive() and not self.workQueue.empty():
                self.workers.append(worker)  # 重新加入线程池中
        #logging.info('All jobs were complete.')

    def add_job(self, callable, *args, **kwds):
        self.workQueue.put((callable, args, kwds))  # 向工作队列中加入请求

    def get_result(self, *args, **kwds):
        return self.resultQueue.get(*args, **kwds)

def check_proxies(ip,port):
    """
    检测代理存活率
    分别访问v2ex.com以及guokr.com
    """
    proxies={'http': 'http://'+str(ip)+':'+str(port)}
    try:
        r0 = requests.get('http://v2ex.com', proxies=proxies,timeout=30,verify=False)
        r1 = requests.get('http://www.guokr.com', proxies=proxies,timeout=30,verify=False)

        if r0.status_code == requests.codes.ok and r1.status_code == requests.codes.ok and "09043258" in r1.content and "15015613" in r0.content:
            #r0.status_code == requests.codes.ok and r1.status_code == requests.codes.ok and 
            print ip,port
            return True
        else:
            return False

    except Exception, e:
        pass
        #sys.stderr.write(str(e))
        #sys.stderr.write(str(ip)+"\t"+str(port)+"\terror\r\n")
        return False

def get_ip181_proxies():
    """
    http://www.ip181.com/获取HTTP代理
    """
    proxy_list = []
    try:
        html_page = requests.get('http://www.ip181.com/',timeout=60,verify=False,allow_redirects=False).content.decode('gb2312')
        jq = PyQuery(html_page)
        for tr in jq("tr"):
            element = [PyQuery(td).text() for td in PyQuery(tr)("td")]
            if 'HTTP' not in element[3]:
                continue

            result = re.search(r'\d+\.\d+', element[4], re.UNICODE)
            if result and float(result.group()) > 5:
                continue
            #print element[0],element[1]
            proxy_list.append((element[0], element[1]))
    except Exception, e:
        sys.stderr.write(str(e))
        pass

    return proxy_list

def get_kuaidaili_proxies():
    """
    http://www.kuaidaili.com/获取HTTP代理
    """
    proxy_list = []
    for m in ['inha', 'intr', 'outha', 'outtr']:
        try:
            html_page = requests.get('http://www.kuaidaili.com/free/'+m,timeout=60,verify=False,allow_redirects=False).content.decode('utf-8')
            patterns = re.findall(r'(?P<ip>(?:\d{1,3}\.){3}\d{1,3})</td>\n?\s*<td.*?>\s*(?P<port>\d{1,4})',html_page)
            for element in patterns:
                #print element[0],element[1]
                proxy_list.append((element[0], element[1]))
        except Exception, e:
            sys.stderr.write(str(e))
            pass

    for n in range(0,11):
        try:
            html_page = requests.get('http://www.kuaidaili.com/proxylist/'+str(n)+'/',timeout=60,verify=False,allow_redirects=False).content.decode('utf-8')
            patterns = re.findall(r'(?P<ip>(?:\d{1,3}\.){3}\d{1,3})</td>\n?\s*<td.*?>\s*(?P<port>\d{1,4})',html_page)
            for element in patterns:
                #print element[0],element[1]
                proxy_list.append((element[0], element[1]))
        except Exception, e:
            sys.stderr.write(str(e))
            pass

    return proxy_list

def get_66ip_proxies():
    """
    http://www.66ip.com/ api接口获取HTTP代理
    """
    urllists = [
        'http://www.proxylists.net/http_highanon.txt',
        'http://www.proxylists.net/http.txt',
        'http://www.66ip.cn/nmtq.php?getnum=1000&anonymoustype=%s&proxytype=2&api=66ip',
        'http://www.66ip.cn/mo.php?sxb=&tqsl=100&port=&export=&ktip=&sxa=&submit=%CC%E1++%C8%A1'
        ]
    proxy_list = []
    for url in urllists:
        try:
            html_page = requests.get(url,timeout=60,verify=False,allow_redirects=False).content.decode('gb2312')
            patterns = re.findall(r'((?:\d{1,3}\.){1,3}\d{1,3}):([1-9]\d*)',html_page)
            for element in patterns:
                #print element[0],element[1]
                proxy_list.append((element[0], element[1]))
        except Exception, e:
            sys.stderr.write(str(e))
            pass

    return proxy_list


def get_proxy_sites():
    wm = WorkManager(20)
    proxysites = []
    proxysites.extend(get_ip181_proxies())
    proxysites.extend(get_kuaidaili_proxies())
    proxysites.extend(get_66ip_proxies())

    for element in proxysites:
        wm.add_job(check_proxies,str(element[0]),str(element[1]))
    wm.start()
    wm.wait_for_complete()


if __name__ == '__main__':
    try:
        get_proxy_sites()
    except Exception as exc:
        print(exc)

Arachni 相关

发布时间:June 21, 2016 // 分类:运维工作,开发笔记,工作日志,linux,代码学习,python // 2 Comments

#! /usr/bin/env python
# -*- coding: utf-8 -*-
import os
import re
import sys
import time
import json
import random
import base64
import hashlib
import threading
import subprocess
from gevent.pool import Pool
from urlparse import urlparse
from get_form import auto_fill_form,getform
"""
这里的作用就是把爬虫结果转化为json
检测思路
1.执行命令
arachni --audit-links --audit-forms --http-user-agent="Mozilla/5.0 (X11; Linux i686; U;) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36" \
    http://testphp.vulnweb.com/ --checks=sql_injection,xss,directory_listing,csrf,xpath_injection --report-save-path=/Users/saline/tools/mycode/spider/report/last0.afr

ruby /Users/saline/tool/tools/arachni/bin/../system/arachni-ui-web/bin/arachni --audit-links --audit-forms --audit-jsons --audit-xmls --audit-ui-inputs --scope-exclude-file-extensions=html --scope-exclude-file-extensions=shtml --http-user-agent="iaskspider/2.0(+http://iask.com/help/help_index.html)" --checks=sql_injection,rfi,directory_listing --report-save-path=/Users/saline/tools/mycode/spider/report/demo.aisec.cn_104446e2321d31be6031ec6daad80c47.afr --timeout=8:00:00 http://demo.aisec.cn/demo/

2.把afr利用arachni_reporter转化为json
#arachni_reporter --reporters-list
arachni_reporter /Users/saline/tools/mycode/spider/report/last0.afr --reporter=json:outfile=/Users/saline/tools/mycode/spider/report/last0_result.json

3.从json里面读取相关的结果,再进行二次分析利用
dist = open('/Users/saline/tools/mycode/spider/report/baimao.json').read()
result = json.loads(dist)
for url in result["sitemap"]:
    if int(result["sitemap"][url]) != 404:
        #输出非404的结果,其实还应该执行对比
        #print url
for urls in result["issues"]:
    print urls["vector"]["action"]+"\t"+urls["vector"]["method"]
    print urls["vector"]["inputs"]

参见帮助文档
http://doc.0xa.cc/r/FIdMhkWFYUvhdKOQQFWtBOltIGxlgsqByLSSPqzkXYRULiYZgm:mobile
http://www.cnblogs.com/vamei/archive/2012/09/23/2698014.html
"""
# 需额外安装arachni
# Arachni rpc clint scan class
class Arachni_Console(object):

    def random_useragent(self):
        USER_AGENTS = [
            "Baiduspider+(+http://www.baidu.com/search/spider.htm)",
            "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
            "Googlebot/2.1 (+http://www.googlebot.com/bot.html)",
            "Googlebot/2.1 (+http://www.google.com/bot.html)",
            "Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)",
            "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)",
            "iaskspider/2.0(+http://iask.com/help/help_index.html)",
            "Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)",
            "Sogou Push Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)",
            "Mozilla/5.0 (compatible; YodaoBot/1.0;http://www.yodao.com/help/webmaster/spider/;)",
            "msnbot/1.0 (+http://search.msn.com/msnbot.htm)",
            "Sosospider+(+http://help.soso.com/webspider.htm)",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11)  Firefox/1.5.0.11; 360Spider",
            "Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/”; )",
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:46.0) Gecko/20100101 Firefox/46.0",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36",
            "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0;  Trident/5.0)",
            "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52",
            "Mozilla/5.0 (X11; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0",
        ]
        return random.choice(USER_AGENTS)
    #扫描中需要注意的是几个地方。允许自定义,交互性较好
    #1.自定义cookie --http-cookie-string
    #2.带有401认证的 --http-authentication-username=username --http-authentication-password=password
    #3.自定义扫描路径 --scope-extend-paths
    #4.自定义ua --http-user-agent
    #5.线程数量 --http-request-concurrency 默认20
    #默认排除html/shtml这类静态文件,可能会对部分jsp的页面带来影响
    def __init__(self, url, http_agent="", cookies=""):
        self.http_agent = "%s"%(self.random_useragent())
        self.start_time         = str(time.time())
        self.url                = url
        self.report             = "%s_%s" % (urlparse(url).netloc, hashlib.md5(self.start_time).hexdigest())
        self.arachni_client  = '/Users/saline/tool/tools/arachni/bin/arachni'
        self.arachni_reporter  = '/Users/saline/tool/tools/arachni/bin/arachni_reporter'
        self.report_file  =  " --report-save-path=/Users/saline/tools/mycode/spider/report/%s.afr" % self.report
        self.cookies  = cookies
        #self.audit = "--audit-links --audit-forms --audit-cookies"
        self.audit = "--audit-links --audit-forms --audit-jsons --audit-xmls --audit-ui-inputs --scope-exclude-file-extensions=html --scope-exclude-file-extensions=shtml"
        self.h_agent = " --http-user-agent=\"%s\"" % (self.http_agent)
        self.h_cookies = " --http-cookie-string=\"%s\"" % (self.cookies)
        self.checks = " --checks=sql_injection,rfi,directory_listing"
        # self.checks = "--checks=rfi,directory_listing,sql_injection,sql_injection_timing,sql_injection_differential,source_code_disclosure,file_inclusion"
        self.timeout = " --timeout=%s" % "8:00:00"
        self.option = self.audit + self.h_agent + self.checks + self.report_file + self.timeout
        self.is_timeout = False
        self.proc       = None
        self.report_jsfile  = '/Users/saline/tools/mycode/spider/report/%s.json' % self.report
        self.result = None

    # Start to Scan
    def _Scan(self):
        # subprocess command
        arachni_cmd = "%s %s %s"%(self.arachni_client,self.option,self.url)
        #self.timer = threading.Timer(6000 * 10 * 10, self.set_time_out())
        #self.timer.start()
        os.system(arachni_cmd)
        #调用subprocess执行有问题。放弃,由于这只是需要结果。所以无需回显
        #self.proc = subprocess.Popen(self.cmd, shell=False)
        #self.proc = subprocess.Popen(arachni_cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        #self.proc.wait()
        #for lines in proc.stdout.readlines():
        #    print(lines)
        #self.timer.cancel()
        #for lines in self.proc.stdout.readlines():

    # timeout function
    def set_time_out(self):
        if self.proc is not None:
            self.is_timeout = True
            self.timer.cancel()
            self.proc.kill()

    def get_report(self):
        # arachni_reporter /tmp/test.afr --report=json:outfile=/tmp/test.json
        try:
            self._Scan()
            self._report()
        except Exception, e:
            pass

        return self.result

    # get result, format is json
    def _report(self):
        self.cmd = [
            self.arachni_reporter,
            "/Users/saline/tools/mycode/spider/report/%s.afr" % self.report,
            '--report=json:outfile=%s' % self.report_jsfile
        ]
        self.proc = subprocess.Popen(self.cmd)
        self.proc.wait()
        #self.result = open(self.report_jsfile).read()
        # del report files
        delcmd = "rm -rf /Users/saline/tools/mycode/spider/report/%s.afr" % self.report
        os.system(delcmd)
        self.result = self.report_jsfile
        #self.result = self.get_json(self.report_jsfile)
        #if len(self.result)>0:
        #    return self.result

        #os.remove(self.report_file)
        #os.remove(self.report_jsfile)
#解析json
def get_json(jsonfile):
    #print int(time.time())
    vul_results = []
    jsonresult = []
    dist = open(jsonfile).read()
    result = json.loads(dist)
    #for url in result["sitemap"]:
    #    if int(result["sitemap"][url]) != 404:
    #        pass
            #print url
    if len(result["issues"])>0:
        for urls in result["issues"]:
            data = ''
            acturl = urls["vector"]["action"]
            #urls.append(str(urls["vector"]["action"]))
            #获取input信息
            for vuln in urls["vector"]["inputs"]:
                if len(auto_fill_form(str(vuln)))>0:
                    value = auto_fill_form(str(vuln))
                    data = data + vuln+'='+value+'&'
                else:
                    value = 'casterjs'
                    data = data + vuln +'='+value+'&'
            #获取到actmethod
            if str(urls["vector"]["method"]).find('get')!=-1:
                actmethod = 'GET'
            elif str(urls["vector"]["method"]).find('post')!=-1:
                actmethod = 'POST'

            if str(actmethod).find('get')!=-1 or str(actmethod).find('GET')!=-1:
                if acturl.find('?') ==-1:
                    acturl = acturl +'?'+data.rstrip('&')
                else:
                    acturl = acturl +'&'+data.rstrip('&')
            if len(data.rstrip('&')) == 0:
                actmethod = 'GET'
            vul_results.append(({"url": acturl,
                "probe": {
                    "payload": data.rstrip('&'),
                    "method": actmethod,
                    "url": acturl,
                    "headers": urls["request"]["headers"],}}))
    if len(result["sitemap"])>0:
        for url in result["sitemap"]:
            if result["sitemap"][url] != 404:
                results = getform(url)
                if result is not None:
                    for lists in results:
                        if lists["probe"]['url'] not in jsonresult:
                            data = base64.b64encode(json.dumps(lists["probe"]))
                            newurl = lists["probe"]['url']
                            jsonresult.append(newurl + ' '+ data)
                            #urls.append(newurl + ' '+ data)

    if vul_results is not None:
        for lists in vul_results:
            if lists["probe"]['url'] not in jsonresult:
                data = base64.b64encode(json.dumps(lists["probe"]))
                newurl = lists["probe"]['url']
                jsonresult.append(newurl + ' '+ data)

    if len(jsonresult)>0:
        return jsonresult

if __name__ == '__main__':
    #domain ="http://0cx.cc/"
    domains = ['http://demo.aisec.cn/demo','http://testphp.vulnweb.com/']
    for domain in domains:
        arachni_console = Arachni_Console(domain, http_agent='')
        try:
            results = get_json(arachni_console.get_report())
            for resu in results:
                print resu
        except Exception as e:
            print(str(e))

 

拾取表单的脚本参考http://0cx.cc/get_form_name.jspx

#!/usr/bin/env python
# -*- encoding: utf-8 -*-

#https://github.com/Arachni/arachni/wiki/REST-server
#https://github.com/Arachni/arachni/wiki/REST-API
'''
开启api
arachni_rest_server 
[开启认证]
(./bin/arachni_rest_server  --address=192.168.87.134 --port=7331  --authentication-username=admin --authentication-password=adminpassword)


1.查看扫描状态
GET /scans

2.提交扫描
POST /scans
json.dumps(xxx.json)
其实需要提供的是url和profiles

3.查看某个id的状态
GET /scans/:id

状态大约有几种[
   a.ready 准备中。但是不曾启动扫描
   b.preparing  准备好了,随时可以启动扫描(即初始化插件)
   c.scanning   扫描中
   d.pausing   扫描被暂停了
   e.paused    扫描已经被停职了
   f.cleanup   扫描已经被中止(即等待插件完成等)
   g.aborted   扫描非正常状态结束
   h.done      扫描结束
]

4.暂停扫描
PUT /scans/:id/pause

5.开始[已暂停的]扫描
PUT /scans/:id/resume

6.提取扫描报告
GET /scans/:id/report
GET /scans/:id/report.json
GET /scans/:id/report.xml
GET /scans/:id/report.yaml
GET /scans/:id/report.html.zip

7.删除扫描
DELETE /scans/:id

'''

import urllib2
import json

class ArachniClient(object):

   with open('./profiles/default.json') as f:
      default_profile = json.load(f)

   def __init__(self, arachni_url = 'http://192.168.87.134:7331'):
      self.arachni_url = arachni_url
      self.options = ArachniClient.default_profile

   def get_http_request(self, api_path):
      return urllib2.urlopen(self.arachni_url + api_path).read()

   def post_api(self, api_path):
      options = json.dumps(self.options)
      request = urllib2.Request(self.arachni_url + api_path, options)
      request.add_header('Content-Type', 'application/json')
      return urllib2.urlopen(request).read()

   def put_request(self, api_path):
      request = urllib2.Request(self.arachni_url + api_path)
      request.get_method = lambda: 'PUT'
      return urllib2.urlopen(request).read()

   def delete_request(self, api_path):
      request = urllib2.Request(self.arachni_url + api_path)
      request.get_method = lambda: 'DELETE'
      return urllib2.urlopen(request).read()
   #获取扫描    
   def get_scans(self):
      return json.loads(self.get_http_request('/scans'))
   #获取扫描状态
   def get_status(self, scan_id):
      return json.loads(self.get_http_request('/scans/' + scan_id))
   #暂停扫描
   def pause_scan(self, scan_id):
      return self.put_request('/scans/' + scan_id + '/pause')
   #重启扫描
   def resume_scan(self, scan_id):
      return self.put_request('/scans/' + scan_id + '/resume')
   #获取扫描结果
   def get_report(self, scan_id, report_format = None):
      if self.get_status(scan_id)['status'] == 'done':

         if report_format == 'html':
            report_format = 'html.zip'

         if report_format in ['json', 'xml', 'yaml', 'html.zip']:
            return self.get_http_request('/scans/' + scan_id + '/report.' + report_format)
         elif report_format == None:
            return self.get_http_request('/scans/' + scan_id + '/report')
         else:
            print 'your requested format is not available.'

      else:
         print 'your requested scan is in progress.'
   #删除扫描
   def delete_scan(self, scan_id):
      return self.delete_request('/scans/' + scan_id)
   #开启扫描
   def start_scan(self):
      if self.options['url']:
         return json.loads(self.post_api('/scans'))
      else:
         print 'Target is not set!'

   def target(self, target_url):
      try:
         urllib2.urlopen(target_url)
         self.options['url'] = target_url
      except urllib2.HTTPError, e:
         print e.code

   def profile(self, profile_path):
      with open(profile_path) as f:
         self.options = json.load(f)

if __name__ == '__main__':
   a = ArachniClient()
   a.profile('./profiles/default.json')
   #'http://testphp.vulnweb.com/','http://23.88.112.156/xvwa/'
   a.target('http://23.88.112.156/xvwa/')
   print a.start_scan()

 

Python识别验证码

发布时间:May 19, 2016 // 分类:开发笔记,代码学习,linux,python,生活琐事 // 1 Comment

最近学习Python识别验证码。

基本都是基于图片处理选择PIL来围绕拓展的。想到之前以前搞过的openvc是否可以做的全面一些呢

下面的是剽窃一个验证码识别大赛的东西,

验证码大概为6种

type1.

type2.

type3.

type4.

type5.

type6.

所有验证码都遵循先分割再识别的流程,分割主要方法是等距分割:type1字符位置固定,分割较简单;type2,type4由于字符数目固定,可以直接等距分割;type3,type5,type6由于字符数目并不固定,所以第一步是识别验证码所包含的字符数,然后再等距分割。汉字/字母/数字的识别模型都是卷积神经网络。type1由于训练集无法直接使用,所以我们根据常用汉字训练了一个通用汉字/拼音识别模型,其他类型验证码均可使用训练集来训练识别模型,其中type3和type4还有可利用的语言上下文信息。
关键技术:当字符数目不确定时,先预测验证码包含的字符个数,再根据字符数进行分割,这样type5,type6的识别问题就分别和type2,type4类似了,这是一种简单通用的验证码识别方案。

算法总体思路如下图,先分割再识别。分割采用等距分割思想,识别采用CNNs模型

type1

此类验证码上方九宫格部分由于字符位置固定,可以直接定位识别。下方3个汉字和一个拼音,垂直方向位置也是固定的,只是水平方向结束的位置在变化,可以按列相加的方法确定结束的位置,然后同时从左右两端开始识别汉字并且确定和上方九宫格汉字的对应关系,找到相似度最大的三个汉字后,剩下的位置就是拼音,直接识别拼音再次到九宫格寻找对应汉字即可。共包含两种识别模型,汉字识别:挑选常用4800个汉字,拼音识别:所有408个拼音。

type2

如图所示,此类验证码包含26个字母和10个数字,而且每张验证码均包含5个位置固定的字符,间距变化很小,所以直接等距分割(相邻字符有重叠)。利用训练集共10万个字符直接训练CNN即可,无需重新生成训练数据。

type3

此类验证码分两种情况。
1.拼音首字母,要求输出4-5个字符的成语或诗句的拼音首字母。根据训练集统计可知共包含约700个不同成语或诗句,所以首先将训练集的首字母重新标注为成语或诗句,接下来就只是一个汉字识别问题。从图像上看,根据“的拼音首字母”的“的”字可以将图像分为左右两部分,所要识别的只是左边部分的4或5个汉字。所以识别流程如下:
1) 根据水平方向从左至右第85像素将图像分割为两部分
2) 预测左边图像部分包含字符的个数n={4,5},是个二分类问题,这一步实验中识别率达到0.999以上
3) 确定最左边字符的起始位置后,根据n的大小,可以在水平方向上进行等距分割
4) 由于垂直方向字符位置有较大变化,所以去除背景像素后,计算每列像素的重心,依次可以进一步确定每个字符在垂直方向的精确位置
5) 分割出来之后,用训练集所训练的汉字识别模型进行识别。测试时,如果所识别出的成语或诗句未在训练集出现,则在训练集中寻找最相似的代替
2.成语。要求直接输出成语本身,这种验证码与输出首字母的验证码类似,但是只需要3) 4) 5)三步即可。

type4

识别思路与type3的第2种情况完全相同,只是数据集不同而已。

type5

统计训练集可知,每个验证码只包含4-5个字母或数字,相邻字符粘连在一起。

  1. 水平方向采用按列相加的方法可以确定起始和结束的位置,垂直方向字符都在第1至36像素间,据此可以从整幅图像中获得验证码的具体位置
  2. 预测验证码包含的字符数之后进行等距分割并识别得到最终结果

###type6

识别思路完全和type5相同,只是字母数字变为了汉字。这类验证码是带噪声的扭曲汉字,而且每个验证码的汉字是随机的、没有任何语义信息,是所有类别中最难的一类。开始我也搁置了很久,但是在某一天偶然得到灵感:就是先预测验证码中包含的字符数,这样一来除了字符识别起来更难一些之外,type6和其他类并没有什么其区别了,所以我在做type5和type6时使用了完全相同的思路和算法。

Mac 下安装 Python-OpenCV

首先确保已经安装了Python.Mac 下可以直接使用 brew 来安装OpenCV,具体步骤如下:

# add opencv
brew tap homebrew/science

# install opencv
brew install opencv

安装必要的python库

pip install numpy
pip install matplotlib

测试是否安装成功

import cv2
import numpy as np
from matplotlib import pyplot as plt

img = cv2.imread('road.png', 0)
plt.imshow(img, cmap='gray', interpolation='bicubic')
plt.xticks([]), plt.yticks([]) # to hide tick values on X and Y axis
plt.show()

作为一个曾经编译过无数次 OpenCV 最后好不容易成功的人来说,我觉得安装 OpenCV 这个问题可以归为玄学,尤其是在 Windows 下,所以安装的时候最好多请教 Google, Good Luck!还是按照这个来吧.Install OpenCV On Mac

具体的内容等我找到资料了慢慢的填充

参考几篇文章

http://aducode.github.io/posts/2014-07-08/python_identification_verification_code.html
http://sineyuan.github.io/2016/01/06/python-captcha/
http://www.pythonclub.org/project/captcha/python-pil
http://bbs.pkbigdata.com/topic/8726c0abb99142c88b8b422483aedb75.html
http://blog.csdn.net/niuwei22007/article/category/5868745
http://www.pyimagesearch.com/2015/06/15/install-opencv-3-0-and-python-2-7-on-osx/
http://seisman.info/anaconda-scientific-python-distribution.html

python multiprocessing apply_async only uses one process

发布时间:April 20, 2016 // 分类:开发笔记,python,生活琐事 // No Comments

某天写了一个多进程的东西。发现无法准确获得返回结果

import os
from multiprocessing import Pool

results = []
def testFunc(files):
    for file in files:
        print "Working in Process #%d" % (os.getpid())
        #This is just an illustration of some logic. This is not what I'm actually doing.
        for line in file:
            if 'dog' in line:
                results.append(line)

if __name__=="__main__":
    p = Pool(processes=2)
    files = ['/path/to/file1.txt', '/path/to/file2.txt']
    results = p.apply_async(testFunc, args = (files,))
    results2 = results.get()
  • apply_async farms out one task to the pool. You would need to call apply_async many times to exercise more processors.
  • Don't allow both processes to try to write to the same list, results. Since the pool workers are separate processes, the two won't be writing to the same list. One way to work around this is to use an ouput Queue. You could set it up yourself, or use apply_async's callback to setup the Queue for you. apply_async will call the callback once the function completes.
  • You could use map_async instead of apply_async, but then you'd get a list of lists, which you'd then have to flatten.

然后更改为

import os
import multiprocessing as mp

results = []   

def testFunc(file):
    result = []
    print "Working in Process #%d" % (os.getpid())
    # This is just an illustration of some logic. This is not what I'm
    # actually doing.
    with open(file, 'r') as f:
        for line in f:
            if 'dog' in line:
                result.append(line)
    return result


def collect_results(result):
    results.extend(result)

if __name__ == "__main__":
    p = mp.Pool(processes=2)
    files = ['/path/to/file1.txt', '/path/to/file2.txt']
    for f in files:
        p.apply_async(testFunc, args=(f, ), callback=collect_results)
    p.close()
    p.join()
    print(results)

再有

import os
from multiprocessing import Pool

results = []
def testFunc(file):
    message =  ("Working in Process #%d" % (os.getpid()))
    #This is just an illustration of some logic. This is not what I'm actually doing.
    for line in file:
        if 'dog' in line:
            results.append(line)
    return message

if __name__=="__main__":
    print("saddsf")
    p = Pool(processes=2)
    files = ['/path/to/file1.txt', '/path/to/file2.txt']
    results = p.map_async(testFunc, files)
    print(results.get())

感觉第二种方式靠谱一些

中间件漏洞检测框架(F-MiddlewareScan)

发布时间:March 20, 2016 // 分类:开发笔记,工作日志,linux,python,windows,生活琐事 // 1 Comment

纯python编写的轻量级中间件漏洞检测框架,实现针对中间件的自动化检测,端口探测->中间件识别->漏洞检测->获取webshell 
参数说明 
-h 必须输入的参数,支持ip(192.168.1.1),ip段(192.168.1),ip范围指定(192.168.1.1-192.168.1.254),最多限制一次可扫描65535个IP。 
-p 指定要扫描端口列表,多个端口使用,隔开 例如:7001,8080,9999。未指定即使用内置默认端口进行扫描(80,4848,7001,7002,8000,8001,8080,8081,8888,9999,9043,9080) 
-m 指定线程数量 默认100线程 
-t 指定HTTP请求超时时间,默认为10秒,端口扫描超时为值的1/2。 
 

漏洞检测脚本以插件形式存在,可以自定义添加修改漏洞插件,存放于plugins目录,插件标准非常简单,只需对传入的IP,端口,超时进行操作,成功返回“YES|要打印出来的信息”即可。 
新增插件需要在 plugin_config.ini配置文件中新增关联(多个漏洞插件以逗号隔开)。 
中间件识别在discern_config.ini文件中配置(支持文件内容和header识别) 

目前内置了19个漏洞插件,希望大家可以一起编写更多的插件,目前还缺少weblogic自动部署和反序列化探测以及中间件的反序列化自动获取webshell的插件等等。 

周末感冒无事,除了吃药意外就是发呆了。好友说想要修改一下,增加CMS识别以及同服查询的功能。动手开始做

def exploit(URL, Thread):
    w = cms.WhatWeb(URL, Thread)
    w.run()
    if w.result:
        return w.result

def whatcms(scan_type,task_host,task_port):
    task_port = '80'
    if task_host.find('http') == -1:
        URL = 'http://'+str(task_host)
    elif task_host.find('///') !=1 and task_host.find('~') == -1:
        URL = str(task_host.replace('///','://'))
    elif task_host.find('///') !=1 and task_host.find('~') != -1:
        URL = task_host.replace('///','://').replace('~',':').rstrip('/')
    log(scan_type,URL,task_port)
    Thread = 40
    try:
        r = requests.get(URL, timeout=15, verify=False)
        if r.status_code == 200:
            return exploit(URL, Thread)
    except Exception as e:
        #print str(e)
        return

def ip2host_get(scan_type,host,port):
    ip2hosts = []
    try:
        req=requests.get('http://www.ip2hosts.com/search.php?ip='+str(host), timeout=45, verify=False)
        src=req.content
        if src.find('result') != -1:
            result = json.loads(src)['result']
            ip = json.loads(src)['ip']
            if len(result)>0:
                for item in result:
                    if len(item)>0:
                        #log(scan_type,host,port,str(item))
                        ip2hosts.append(item.replace('://','///').replace(':','~'))
    except Exception, e:
        print str(e)
        pass
    return ip2hosts

再次修改了其中的顺序,

    def run(self):
        while True:
            queue_task = self.queue.get()
            task_type,task_host,task_port = queue_task.split(":")
            if task_type == 'portscan':
                port_status = scan_port(task_type,task_host,task_port)
                if port_status == True:
                    #如果端口开发,推送到任务
                    queue.put(":".join(['ip2host_get',task_host,task_port]))
            elif task_type == 'ip2host_get':
                #针对存货IP发起旁站查询
                result = []
                urls = ip2host_get(task_type,task_host,task_port)
                #queue.put(":".join(['discern',task_host,task_port]))
                urls.insert(0,task_host)
                result.extend(urls)
                urls = list(set(result))
                if len(urls)>0:
                    #list can not use find
                    for url in urls:
                        if len(url)>0:
                            #print url
                            #put url in queue,but some qestion in Threads and queue
                            queue.put(":".join(['whatcms',str(url),task_port]))
            elif task_type == 'whatcms':
                cms = whatcms(task_type,task_host,task_port)
                queue.put(":".join(['discern',task_host,task_port]))
                if cms == None:
                    "go on 但是没什么乱用"
                    #以后增加插件式扫描

            elif task_type == 'discern':
                #针对中间件的识别
                discern_type = scan_discern(task_type,task_host,task_port)
                if discern_type:
                    queue.put(":".join([discern_type,task_host,task_port]))
            else:
                scan_vul(task_type,task_host,task_port)
            self.queue.task_done()

但是问题来了,线程经常性的奔溃掉,然后就无奈了

然后发现了一个有意思的东西https://raw.githubusercontent.com/erevus-cn/pocscan/master/web/tasks.py

# coding:utf-8
import gevent
from gevent.pool import Pool
from web.lib.utils import *
from pocscan.poc_launcher import Poc_Launcher
from celery import Celery, platforms

app = Celery()

# 允许celery以root权限启动
platforms.C_FORCE_ROOT = True

# 修改celery的全局配置
app.conf.update(
    CELERY_IMPORTS = ("tasks", ),
    BROKER_URL = 'amqp://guest:guest@localhost:5672/',
    CELERY_RESULT_BACKEND = 'db+mysql://root:123456@127.0.0.1:3306/pocscan',
    CELERY_TASK_SERIALIZER='json',
    CELERY_RESULT_SERIALIZER='json',
    CELERY_TIMEZONE='Asia/Shanghai',
    CELERY_ENABLE_UTC=True,
    BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600}, # 如果任务没有在 可见性超时 内确认接收,任务会被重新委派给另一个Worker并执行  默认1 hour.
    CELERYD_CONCURRENCY = 50 ,
    CELERY_TASK_RESULT_EXPIRES = 1200,  # celery任务执行结果的超时时间,我的任务都不需要返回结
    # BROKER_TRANSPORT_OPTIONS = {'fanout_prefix': True},       # 设置一个传输选项来给消息加上前缀
)

# 失败任务重启休眠时间300秒,最大重试次数5次
#@app.task(bind=True, default_retry_delay=300, max_retries=5)
@app.task(time_limit=3600)
def run_task_in_gevent(url_list, poc_file_dict):     # url_list 每个进程分配到一定量的url
    poc = Poc_Launcher()
    pool = Pool(100)
    for target in url_list:
        for plugin_type,poc_files in poc_file_dict.iteritems():
            for poc_file in poc_files:
                if target and poc_file:
                    target = fix_target(target)
                    pool.add(gevent.spawn(poc.poc_verify, target, plugin_type, poc_file))
    pool.join()

搜了下Celery,专门用于解决任务队列用于分发工作给不同线程。回头研究下

 

参考文章:

http://docs.jinkan.org/docs/celery/
http://my.oschina.net/u/2306127/blog/417360
http://rfyiamcool.blog.51cto.com/1030776/1325062
http://www.tuicool.com/articles/qi6Nve

MacOSX安装autopy时遇到错误

发布时间:March 2, 2016 // 分类:运维工作,开发笔记,工作日志,转帖文章,python,windows // No Comments

spynner是一个QtWebKit的客户端,它可以模拟浏览器,完成加载页面、引发事件、填写表单等操作。

这个模块可以在Python的官网找到。

下载地址: https://pypi.python.org/pypi/spynner/2.5

解压后,cd到安装目录,然后输入sudo python configure.py install安装该模块。

这样Spynner模块就安装完成了,在python shell中试试import spynner看看该模块有没有安装完成。

其实是安装spynner的时候遇到的.习惯的使用pip去安装了pip install spynner的时候发现了这个错误

src/screengrab.c:48:26: warning: implicit declaration of function 'CGDisplayBitsPerPixel' is invalid in C99
      [-Wimplicit-function-declaration]
        bitsPerPixel = (uint8_t)CGDisplayBitsPerPixel(displayID);
                                ^
src/screengrab.c:174:15: warning: 'kCGLPFAFullScreen' is deprecated: first deprecated in OS X 10.6 [-Wdeprecated-declarations]
        attribs[0] = kCGLPFAFullScreen;
                     ^
/System/Library/Frameworks/OpenGL.framework/Headers/CGLTypes.h:71:2: note: 'kCGLPFAFullScreen' declared here
        kCGLPFAFullScreen OPENGL_ENUM_DEPRECATED(10_0, 10_6)     =  54,
        ^
src/screengrab.c:191:2: warning: 'CGLSetFullScreen' is deprecated: first deprecated in OS X 10.6 [-Wdeprecated-declarations]
        CGLSetFullScreen(glContext);
        ^
/System/Library/Frameworks/OpenGL.framework/Headers/OpenGL.h:73:17: note: 'CGLSetFullScreen' declared here
extern CGLError CGLSetFullScreen(CGLContextObj ctx) OPENGL_DEPRECATED(10_0, 10_6);
                ^
src/screengrab.c:194:2: warning: implicit declaration of function 'glReadBuffer' is invalid in C99 [-Wimplicit-function-declaration]
        glReadBuffer(GL_FRONT);
        ^
src/screengrab.c:194:15: error: use of undeclared identifier 'GL_FRONT'
        glReadBuffer(GL_FRONT);
                     ^
src/screengrab.c:197:2: warning: implicit declaration of function 'glFinish' is invalid in C99 [-Wimplicit-function-declaration]
        glFinish();
        ^
src/screengrab.c:199:6: warning: implicit declaration of function 'glGetError' is invalid in C99 [-Wimplicit-function-declaration]
        if (glGetError() != GL_NO_ERROR) return NULL;
            ^
src/screengrab.c:199:22: error: use of undeclared identifier 'GL_NO_ERROR'
        if (glGetError() != GL_NO_ERROR) return NULL;
                            ^
src/screengrab.c:207:2: warning: implicit declaration of function 'glPopClientAttrib' is invalid in C99
      [-Wimplicit-function-declaration]
        glPopClientAttrib(); /* Clear attributes previously set. */
        ^
src/screengrab.c:223:2: warning: implicit declaration of function 'glPushClientAttrib' is invalid in C99
      [-Wimplicit-function-declaration]
        glPushClientAttrib(GL_CLIENT_PIXEL_STORE_BIT);
        ^
src/screengrab.c:223:21: error: use of undeclared identifier 'GL_CLIENT_PIXEL_STORE_BIT'
        glPushClientAttrib(GL_CLIENT_PIXEL_STORE_BIT);
                           ^
src/screengrab.c:225:2: warning: implicit declaration of function 'glPixelStorei' is invalid in C99 [-Wimplicit-function-declaration]
        glPixelStorei(GL_PACK_ALIGNMENT, BYTE_ALIGN); /* Force alignment. */
        ^
src/screengrab.c:225:16: error: use of undeclared identifier 'GL_PACK_ALIGNMENT'
        glPixelStorei(GL_PACK_ALIGNMENT, BYTE_ALIGN); /* Force alignment. */
                      ^
src/screengrab.c:226:16: error: use of undeclared identifier 'GL_PACK_ROW_LENGTH'
        glPixelStorei(GL_PACK_ROW_LENGTH, 0);
                      ^
src/screengrab.c:227:16: error: use of undeclared identifier 'GL_PACK_SKIP_ROWS'
        glPixelStorei(GL_PACK_SKIP_ROWS, 0);
                      ^
src/screengrab.c:228:16: error: use of undeclared identifier 'GL_PACK_SKIP_PIXELS'
        glPixelStorei(GL_PACK_SKIP_PIXELS, 0);
                      ^
src/screengrab.c:235:2: warning: implicit declaration of function 'glReadPixels' is invalid in C99 [-Wimplicit-function-declaration]
        glReadPixels(x, y, width, height,
        ^
src/screengrab.c:236:30: error: use of undeclared identifier 'GL_BGRA'
                     MMRGB_IS_BGR ? GL_BGRA : GL_RGBA,
                                    ^
src/screengrab.c:236:40: error: use of undeclared identifier 'GL_RGBA'
                     MMRGB_IS_BGR ? GL_BGRA : GL_RGBA,
                                              ^
10 warnings and 9 errors generated.
error: Setup script exited with error: command 'cc' failed with exit status 1

仔细看了下发现是autopy的错误,查看src/screengrab.c发现是由于OpenGL没有更新版本,存在一些已经被弃用的方法。解决办法很简单,安装libpng,下载地址。安装完成后发现还是继续报错

git clone git://github.com/msanders/autopy.git
cd autopy
python setup.py build
src/png_io.c:3:10: fatal error: 'png.h' file not found  
#include <png.h>  
         ^  
1 error generated.  
error: command 'cc' failed with exit status 1  

解决办法很简单..直接指向就好了


export LIBRARY_PATH="/usr/local/lib:/usr/local/include/libpng"
export C_INCLUDE_PATH="/usr/local/lib:/usr/local/include/libpng"
python setup.py build
sudo python setup.py install