Often there is already some infrastructure to collect notifications about your systems based on SNMP traps. It's quite easy to enable SMF to send SNMP traps when a service transitions to a different state.

In order so send the SNMP traps to a different system, you have to configure a trapsink.
root@master:/#echo 'trapsink 10.0.0.212 public' >>/etc/net-snmp/snmp/snmpd.conf
Now you have to enable net-snmp and snmp-notify
root@master:/# svcs enable net-snmp
root@master:/# svcadm enable svc:/system/fm/snmp-notify:default
If you want to enable SNMP traps for all services you can configure it globally.
root@master:/# svccfg setnotify -g from-online,to-maintenance snmp:active
You can check this via listnotify:
root@master:/# svccfg listnotify -g 
    Event: to-maintenance (source: svc:/system/svc/global:default)
        Notification Type: snmp
            Active: true

    Event: from-online (source: svc:/system/svc/global:default)
        Notification Type: snmp
            Active: true
If you want to enable SNMP for a single service you can configure it globally:
svccfg setnotify -g from-online,to-maintenance snmp:active
Let's check this again:
root@master:/# svccfg -s svc:/milestone/goals:default listnotify
    Event: to-maintenance (source: svc:/milestone/goals:default)
        Notification Type: snmp
            Active: true

    Event: from-online (source: svc:/milestone/goals:default)
        Notification Type: snmp
            Active: true

root@master:/#
In order to test this, let us configure sendmail and apache24 into the goal service. However we could do this with every SMF serviice you want. However goal services are a nice feature by and in itself. If one of the services isn't running this service to transition into the maintenance state. Instead of monitoring several services you only monitor the goal services or depend on this goal services.
root@master:/# svcadm enable apache24
root@master:/# svcadm  goals svc:/network/smtp:sendmail svc:/network/http:apache24
Okay, we now disable apache24 and thus force goals into maintenance.
root@master:/# svcadm disable apache24
root@master:/# svcs apache24
STATE          STIME             FMRI
disabled       12:46:17          svc:/network/http:apache24
root@master:/# svcs goals
STATE          STIME             FMRI
maintenance    12:46:17          svc:/milestone/goals:default
Now we can check what is send to the snmpserver. I will just check this via tcpdump
root@debian:/var/log# tcpdump ip host 10.0.0.198
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s3, link-type EN10MB (Ethernet), capture size 262144 bytes
14:46:02.990299 IP 10.0.0.198.49081 > debian.snmp-trap:  Trap(290)  E:sun.2.195.1.7 10.0.0.198 enterpriseSpecific s=1 653212 E:sun.2.195.1.1.1.2.36.51.55.48.102.56.54.102.100.45.56.48.53.53.45.52.101.55.55.45.57.57.102.101.45.99.49.99.55.50.49.101.51.51.100.98.56="370f86fd-8055-4e77-99fe-c1c721e33db8" E:sun.2.195.1.1.1.3.36.51.55.48.102.56.54.102.100.45.56.48.53.53.45.52.101.55.55.45.57.57.102.101.45.99.49.99.55.50.49.101.51.51.100.98.56="FMD-8000-4M" E:sun.2.195.1.1.1.4.36.51.55.48.102.56.54.102.100.45.56.48.53.53.45.52.101.55.55.45.57.57.102.101.45.99.49.99.55.50.49.101.51.51.100.98.56="http://support.oracle.com/msg/FMD-8000-4M"
14:46:02.994349 IP 10.0.0.198.49081 > debian.snmp-trap:  Trap(290)  E:sun.2.195.1.7 10.0.0.198 enterpriseSpecific s=1 653212 E:sun.2.195.1.1.1.2.36.51.55.48.102.56.54.102.100.45.56.48.53.53.45.52.101.55.55.45.57.57.102.101.45.99.49.99.55.50.49.101.51.51.100.98.56="370f86fd-8055-4e77-99fe-c1c721e33db8" E:sun.2.195.1.1.1.3.36.51.55.48.102.56.54.102.100.45.56.48.53.53.45.52.101.55.55.45.57.57.102.101.45.99.49.99.55.50.49.101.51.51.100.98.56="FMD-8000-6U" E:sun.2.195.1.1.1.4.36.51.55.48.102.56.54.102.100.45.56.48.53.53.45.52.101.55.55.45.57.57.102.101.45.99.49.99.55.50.49.101.51.51.100.98.56="http://support.oracle.com/msg/FMD-8000-6U"
14:46:17.040036 IP 10.0.0.198.49081 > debian.snmp-trap:  Trap(365)  E:sun.2.197.1.2 10.0.0.198 enterpriseSpecific s=1 654623 E:sun.2.197.1.1.1.93283196="master" E:sun.2.197.1.1.2.93283196="SMF-8000-SR" E:sun.2.197.1.1.3.93283196="The indicated service has transitioned to the offline state." E:sun.2.197.1.1.4.93283196=07_e4_06_09_0c_2e_11_00_2b_00_00 E:sun.2.197.1.1.5.93283196="svc:/milestone/goals:default" E:sun.2.197.1.1.6.93283196=1 E:sun.2.197.1.1.7.93283196=0 E:sun.2.197.1.1.8.93283196="a dependency activity required a stop"
14:46:17.044718 IP 10.0.0.198.49081 > debian.snmp-trap:  Trap(422)  E:sun.2.197.1.2 10.0.0.198 enterpriseSpecific s=1 654623 E:sun.2.197.1.1.1.93283196="master" E:sun.2.197.1.1.2.93283196="SMF-8000-X2" E:sun.2.197.1.1.3.93283196="The indicated service has transitioned to the maintenance state." E:sun.2.197.1.1.4.93283196=07_e4_06_09_0c_2e_11_00_2b_00_00 E:sun.2.197.1.1.5.93283196="svc:/milestone/goals:default" E:sun.2.197.1.1.6.93283196=0 E:sun.2.197.1.1.7.93283196=4 E:sun.2.197.1.1.8.93283196="goal service has dependencies that cannot be satisfied without administrative intervention"
14:46:17.101144 IP 10.0.0.198.49081 > debian.snmp-trap:  Trap(290)  E:sun.2.195.1.7 10.0.0.198 enterpriseSpecific s=1 654629 E:sun.2.195.1.1.1.2.36.51.97.50.50.49.54.55.98.45.49.48.52.51.45.52.100.57.57.45.97.99.56.102.45.99.54.55.49.52.56.49.51.48.98.52.50="3a22167b-1043-4d99-ac8f-c67148130b42" E:sun.2.195.1.1.1.3.36.51.97.50.50.49.54.55.98.45.49.48.52.51.45.52.100.57.57.45.97.99.56.102.45.99.54.55.49.52.56.49.51.48.98.52.50="SMF-8000-YX" E:sun.2.195.1.1.1.4.36.51.97.50.50.49.54.55.98.45.49.48.52.51.45.52.100.57.57.45.97.99.56.102.45.99.54.55.49.52.56.49.51.48.98.52.50="http://support.oracle.com/msg/SMF-8000-YX"
14:46:17.264339 IP 10.0.0.198.49081 > debian.snmp-trap:  Trap(350)  E:sun.2.197.1.2 10.0.0.198 enterpriseSpecific s=1 654645 E:sun.2.197.1.1.1.93283196="master" E:sun.2.197.1.1.2.93283196="SMF-8000-UQ" E:sun.2.197.1.1.3.93283196="The indicated service has transitioned to the disabled state." E:sun.2.197.1.1.4.93283196=07_e4_06_09_0c_2e_11_00_2b_00_00 E:sun.2.197.1.1.5.93283196="svc:/network/http:apache24" E:sun.2.197.1.1.6.93283196=1 E:sun.2.197.1.1.7.93283196=3 E:sun.2.197.1.1.8.93283196="a disable was requested"
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
root@debian:/var/log
Of course for a full implementation you have to set up a real snmptrap receiver and include the MIBs.

No comments

Add Comment

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA