— Log management as a λrchitecture@2016 OpenSource UseCase#* —
The incontournable logs aggregation Hortonworks, MapR and myDistrib (
2.6.X) has been continued.
Oldschool grep to get into logs.
logstash as my "λ 1 en 2" : #1 elasticsearch, #2 webHDFS.
...
output
{
elasticsearch
{
hosts => ["jbd-vm01:9200"]
}
webhdfs
{
host => "jbd-vm01"
port => "50070"
path => "/test/lstash-00/%{+YYYY-MM-dd}/%{host}/%{hdfs}"
user => "hduser"
codec => line { format => "%{message}" }
}
}
...
#1 Elasticsearch world, plugins Architecture xN.
#2 Grep 2.0 to investigate everywhere, whenwhere, for datascientist only. Hadoop World, Architecture #*xN.
From ASM, C/C++, SDK32/MFC, JAVA, 2D/3D, PHP, Androïd, HTML5 to Pig Latin, jruby, gem by the way Shell(s), PERL, IDL, OCCAM, LISP, PROLOG, APT...
$ gem build logstash-output-webhdfs.gemspec $ sudo gem install logstash-output-webhdfs-2.0.4.gem -f $ gem list --local | grep webhdfs logstash-output-webhdfs (2.0.4) $ logstash-2.3.2/bin/logstash-plugin install logstash-output-webhdfs Validating logstash-output-webhdfs Installing logstash-output-webhdfs Installation successful $ logstash-2.3.2/bin/logstash-plugin list | grep webhdfs webhdfs
and grok to always end with vi, ZZ.
...
filter
{
grok
{
match => { "message" => "%{TIMESTAMP_ISO8601:date} %{LOGLEVEL:level} %{JAVACLASS:class}%{NOTSPACE} %{GREEDYDATA:texte}" }
}
mutate
{
add_field => { "hadoop" => "%{path}" }
add_field => { "hdfs" => "%{path}" }
}
mutate
{
gsub => [ "hadoop", ".*hduser-", "" ]
gsub => [ "hadoop", "-jbd.*$", "" ]
gsub => [ "hdfs", "^.*logs/", "" ]
}
}
... myDistrib 2.6.X network security chalenge continue with VBox tunnings. @ myToolBox 2.7.X
and a few advanced all-road tunningS for Linux-eux.
...
04:33:08.661455 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
...
Status: CORRUPT -> The filesystem under path '/' is HEALTHY
...
WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hduser OPERATION=...Finished - Failed TARGET=RMAppManager RESULT=FAILURE...
...
# @JBD patch logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/java
...
# cf doc !!! message_format => "%{message}"
...
