Организация регулярного дайджеста логов с помощью python и ansible на примере asterisk

Когда я создавал данный инструмент, я не был знаком с logwatch. Мне захотелось видеть ситуацию с логами на своих серверах в целом, и, так я сделал этот велосипед. Думаю, что данный механизм может помочь новичкам в понимании альтернативных возможностей ansible.

Используемые программные продукты:

• python 2.7.14
• ansible 2.3
• сервера asterisk на базе FreePBX 13

Механизм состоит из двух частей – python скрипт, который занимается обработкой лог файла и отправкой отчета на почту, и плейбука для сбора логов с серверов и передачи их скрипту для обработки.

Сам плейбук:

---
- name: parseastlogs
  hosts: production_asterisks
  vars:
    date: "{{ lookup('pipe', 'date +%Y%m%d') }}"
    ipaddr: "{{ ansible_default_ipv4.address }}"
  tasks:
   - debug: var=date
   - debug: var=ipaddr
   - fetch:	 
      src: /var/log/asterisk/full-{{ date }}
      dest: /tmp/full-{{ date }}-{{ ipaddr }}
      flat: yes

   - local_action: "shell /etc/ansible/localscripts/astReporter.py {{ ipaddr }} full-{{ date }}-{{ ipaddr }}"

В плейбуке задаем переменные, ответственные за имя файла и забираем лог файлы с сервера модулем fetch. Так как у нас будут уникальные файлы - используем параметр flat, чтобы избежать длинных путей к лог файлам.

Скрипт:

#!/usr/bin/python2.7
import re
from collections import Counter
import yagmail
import sys
import datetime
import os

servername=sys.argv[1]
yag=yagmail.SMTP(user='ansible@2.1',password=None,host='192.168.2.1',port='25',smtp_starttls=False,smtp_set_debuglevel=0,smtp_skip_login=True)
recipients=['[e-mail]']


filename=sys.argv[2]
workdir='/tmp/'
log_file=open(workdir+filename,'r')#sys.argv[1]
loglist=list()
verbosity=['NOTICE','ERROR','WARNING']
regexes = ["Call from '.*' .* to extension '.*' rejected because extension not found in context '.*'.",
"Identifier \d+, identifier_type \d+ not found in identifier list",
"Invalid result identifier \d+ passed in aMYSQL_clear",
"This function can only be used on SIP channels.",
"fwrite() returned error: Broken pipe",
"CDR requires a value \(CDR\(variable\)=value\)",
"Received SIP subscribe for peer without mailbox: .*",
"Removed interface '.*' from queue '.*'",
"Peer '.*' is trying to register, but not configured as host=dynamic",
"Registration from '.*' failed for '.*' - Peer is not supposed to register",
"Unable to join queue '.*'",
"Attempt to pause interface Local/@from-queue/n, not found",
"PRESENCE_STATE unknown",
"EXTENSION_STATE requires an extension",
"Prodding channel '.*' failed",
"Channel '.*' sent to invalid extension but no invalid handler: context,exten,priority=.*",
"Can't send 10 type frames with PJSIP",
"Attempt to pause interface .+, not found",
"Attempt to unpause interface .+, not found",
"no samples for ulawtolin",
"Could not find matching INVITE transaction for CANCEL request",
"Peer '.*' is now Reachable. \(.*\)",
"Peer '.*' is now UNREACHABLE!  Last qualify: .*",
"Registration from .* failed for '.*' - Wrong password",
"Retransmission timeout reached on transmission .*",
"no samples for alawtolin",
"Peer '.*' is now Lagged. \(\d+ms / \d+ms\)",
"Call completed to .*",
"Invalid retrytime at line \d+ of .*",
"Not accepting call completion offers from call-forward recipient .*",
"No such context '.*' for macro '.*'\. Was called by .*",
"[pP]ickup .* attempt by .*",
"Call failed to go through, reason \(5\) Remote end is Busy",
"Deprecated syntax found\..* ",
"No digits dialed for atxfer.",
"Unable to create channel of type 'SIP' \(cause \d+ - Subscriber absent\)",
"'tls' is not a valid transport type when tlsenable=no\. If no other is specified, the defaults from general will be used\.",
"'tcp' is not a valid transport type when tcpenable=no\. If no other is specified, the defaults from general will be used\.",
"Queued call to .* expired without completion after \d+ attempts",
"Re-invite to non-existing call leg on other UA. SIP dialog .*\. Giving up.",
"Channel .* not found!  Variable 'BLKVM' not set to .*\.",
"Remote host can't match request CANCEL to call .*\. Giving up\.",
"Unable to execute query \[.*\]",
"SQL Exec Direct failed \(-1\)!\[",
"SQL Execute returned an error .*",
"   -- Re-registration for .*",
"Outbound Registration\: Expiry for .* is \d+ sec \(Scheduling reregistration in \d+ s\)",
"Correct auth, but based on stale nonce received from '.*'",
"Unable to write frametype: 2",
"Received response: \"Forbidden\" from '\".*\" .*'",
"Huh\?  Not an RDNIS SIP header .*",
"Hanging up call .* - no reply to our critical packet .*",
"Cancelling retransmit of OPTIONs \(call id .*\)  ",
"Still have a QUALIFY dialog active, deleting",
"The use of '_\.' for an extension is strongly discouraged and can have unexpected behavior.  Please use '_X\.' instead at line .*",
"Context '.*' tries to include nonexistent context '(.*)'",
"aMYSQL_query: mysql_query failed\. Error: Duplicate entry .*",
"RTCP SR transmission error to .*, rtcp halted Operation not permitted",
"Failed to write frame to '.*': Resource temporarily unavailable",
"Unable to forward frametype: 2",
"Timeout on .* on non-critical invite transaction.",
"Unexpected control subclass '\d+'",
"Context '.*' for macro '.*' lacks .*",
"No response received from '.*' on registration attempt to '.*', retrying in '\d+'",
"Unknown RTP codec 90 received from '.*'",
"Invalid extension '.*', but no rule 'i' or 'e' in context '.*'",
"Added interface '.*' to queue '.*'",
"Exceptionally long voice queue length queuing to .*",
"Request from '.*' failed for '.*' \(callid: .*\) - No matching endpoint found",
"Junk at the beginning of frame \d+",
"Unable to register extension at line .*",
"Unable to register extension '.*' priority .* in '.*', already in use",
"Unable to load config file .*",
'Purely numeric hostname \(\d+\), and not a peer--rejecting!',
"Unknown directive .* at line \d+ .*",
"Context '.*' already included in '.*' context on include at line \d+ of .*",
"No closing parenthesis found\? '.*' at line \d+ of .*",
"Can't use '.*' priority on the first entry at line \d+ of .*",
"The .* options are deprecated. Please see UPGRADE.txt for information",
"Can't use '.*' priority on the first entry at line \d+ of .*",
"Call failed to go through, reason .*",
"Unable to open .*",
"Playback of message .* failed",
"File .* does not exist in any format",
"Playback failed on .* for .*",
"Adding .* to .*",
"Can't move channel. One or both is dead .*",
"Unable to complete call pickup of .*",
"Pickup .* failed by .*",
"No entry in voicemail config file for .*",
"   \-\- Registration for '.*' timed out, trying again \(Attempt #[0-9]+\)",
"Disconnecting call .* for lack of RTP activity in [0-9]+ seconds",
"Failed to initialize .* declining image stream",
"Can't send 10 type frames with SIP write",
"Set requires an '=' to be a valid assignment.",
"Timeout but no rule 't' or 'e' in context .*",
"RTCP RR transmission error to .*, rtcp halted Operation not permitted",
"Unable to acquire target extension for attended transfer.",
"Unterminated comment detected beginning on line"]
notmatched=list()
matchedregex=list()

combinedre="("+")|(".join(regexes) + ")"

# данный класс сделан для удобной работы со строками из лог файла. Функционал несколько избыточен и не все поля используются.
class logitem:
    ldatetime=None
    ltype=None
    lchan=None
    lsource=None
    lcontent=None
    def __init__(self,ldate,ltype,lch,lsrc,lcont):
        self.ldatetime=ldate
        self.ltype=ltype
        self.lchan=lch
        self.lsource=lsrc
        self.lcontent=lcont


# данная функция осуществляет работу с лог файлом
def main():

    i=0
    dumpchanstart=False

    for line in log_file:
#игнорируем сообщения отладки Dumpchan
    	if 'app_dumpchan.c' in line:
            dumpchanstart = 0
            for line in log_file:
                if dumpchanstart==2:
                    break

                if "================================================================================" in line:
                    dumpchanstart+=1
#обрабатываем строки со стандартными сообщениями, которые начинаются с [
	if line[0]=="[":

         buf=re.split('] ', line)
# обрабатываем строки с заданной verbosity 
         if any(x in buf[1] for x in verbosity):
                buf1spl=buf[1].split('-')
                channum=''
                buf2spl=re.split('c: ',buf[2])
                if len(buf1spl)>1:
                    channum=buf1spl[1]
                itemcheck=logitem(buf[0][1:],buf[1].split('[')[0],channum,buf2spl[0],buf2spl[1])
                filtration(itemcheck.lcontent)

#функция распределяет получившийся объект в список matchedregex, если он попал под шаблон и в notmatched, если нет.	
def filtration(logitem):

    found = False
    for regexitem in regexes:

        if  re.match(regexitem,logitem):
            found= True
            matchedregex.append(regexitem)
            continue

    if found==False:
        notmatched.append(logitem)


#функция генерации основного тела письма. В блоке if..elif производится цветовое выделение важных сообщений
def genContents():
    content=u"<html>"
    content+=u"<head></head>"
    content+=u"<body>"
    content+=u"<h1>Log digest for " +servername+ u"</h1>"
    content+=u"<table  style='width: 60%;' border = \"1\" cellpadding = \"1\"'>"
    for key, val in Counter(matchedregex).most_common():
        if u"rejected because extension not found in context" in key:
            key=u"<b style='background-color:#e8edff'>"+key+u"</b>"
        elif u"No matching endpoint found" in key:
            key=u"<b style='background-color:#fd6161'>"+key+u"</b>"
        elif u"No closing parenthesis " in key:
            key = u"<b style='background-color:#98862a'>" + key + u"</b>"
        elif u"Wrong password" in key:
            key=u"<b style='background-color:#fd6161'>"+key+u"</b>"    
        elif u"Unterminated comment detected beginning on line" in key:
            key=u"<b style='background-color:#fd6161'>"+key+u"</b>"    
        content += u"<tr>"
        content+= u"<td>"+key+u"</td><td>" +str(val) +u"</td>"
        content += u"</tr>"
    content+=u"</table><br>"
    content+=u"<h1>not matched logitems:</h1><br>"
    for item in notmatched:
        content+=item
    content+=u"</body>\n"
    content+=u"</html>\n"
    return  content

main()
cont=genContents()
yag.send(recipients,subject='Log digest for '+servername,contents=cont)
os.remove(workdiir+filename)

запуск скрипта осуществляется командой: /usr/local/bin/ansible-playbook /etc/ansible/playbooks/parseastlogs.yml

Результатом выполнения команды будет некоторое количество писем(по одному письму на каждый сервер из группы production_asterisk) с примерно следующим содержанием:



Если есть вопросы или предложения, готов на них ответить.