Présentation

Le BigData OpenSource au service du 4001 et des Chiffres relatifs aux crimes et délits enregistrés par les services de police et de gendarmerie.

Milestones

Pig environment.

REGISTER '/data31tech/PiggyBank/contrib/piggybank/java/piggybank.jar'
...
JBD_4001_0_REF = LOAD '/data31tech/input/4001-0-REF.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage(';')  AS ...
...
JBD_4001_0 = LOAD '/data31tech/input/4001-0-2.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage(';') AS (index:chararray...
JBD_4001_1 = LOAD '/data31tech/input/4001-1-4.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage(';') AS (date:datetime...
...

Elasticsearch storage.

REGISTER /data31tech/elasticsearch-hadoop-2.1.1/dist/elasticsearch-hadoop-pig-2.1.1.jar;
...
STORE JBD_4001_0 INTO '4001_0/csv' USING org.elasticsearch.hadoop.pig.EsStorage('...');
STORE JBD_4001_1 INTO '4001_1/csv' USING org.elasticsearch.hadoop.pig.EsStorage('...');
...

Kibana visualisation #1.

Kibana + Elasticsearch visualisation #2.

TODOs

OpenSource & BigData roadmaps.

$ ls | grep "elastic\|kibana\|logstash"
elasticsearch-1.5.2
elasticsearch-1.6.0
elasticsearch-1.7.2
elasticsearch-2.0.0-rc1
elasticsearch-hadoop-2.1.1
kibana-4.0.2-linux-x86
kibana-4.1.0-linux-x86
kibana-4.1.1-linux-x86
kibana-4.1.2-linux-x86
kibana-4.2.0-beta2-linux-x86
logstash-1.5.1
logstash-1.5.2

To be continued...