Alejandro Nolla
Index

Moloch: Erasing Data and Restore Database

   Comments

This post is just a quick tip for restoring moloch state (database and pcap data) to have a superb fresh installed moloch system. I have received and email asking for this, so maybe it could be useful for someone. If you don’t know what moloch is take a look to his github and my previous post to know more about this powerful network traffic capturer and indexer distributed system.

Restoring elasticsearch schema and indexed data

First, we need to erase indexed data and, optionally, also user data, to do this moloch includes a perl script for managing database:

moloch database management options
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

~/moloch/db# ./db.pl

Missing arguments

./db.pl <ESHOST:ESPORT> <command> [<options>]

Commands:
  init                  - Clear ALL elasticsearch moloch data and create schema
  wipe                  - Same as init, but leaves user database untouched
  upgrade               - Upgrade Moloch's schema in elasticsearch from previous versions
  usersexport <fn>      - Save the users info to <fn>
  usersimport <fn>      - Load the users info from <fn>
  rotate <type> <num>   - Perform daily maintenance
       type             - Same as rotateIndex in ini file = daily,weekly,monthly
       num              - number indexes to keep

So, if we want to restore database state - users included - we have to do the following:

restoring moloch’s elastic search data
1
2
3
4
5
6
7
8
9
10
11
12

~/moloch/db# ./db.pl ELASTICSEARCH_IP:9200 init
It is STRONGLY recommended that you stop ALL moloch captures and viewers before proceeding.

There is 1 elastic search node, if you expect more please fix first before proceeding.

It appears this elastic search cluster already has moloch installed, this will delete ALL data in elastic search! (It does not delete the pcap files on disk.)

Type "INIT" to continue - do you want to erase everything?
INIT
Erasing
Creating
Finished.  Have fun!

Now elastic search only have basic schema (with users database restored); to know more about what db.pl have done take a look to his source code:

moloch database management source code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

if ($ARGV[1] =~ /(init|wipe)/) {

    if ($ARGV[1] eq "init" && $main::versionNumber >= 0) {
        print "It appears this elastic search cluster already has moloch installed, this will delete ALL data in elastic search! (It does not delete the pcap files on disk.)\n\n";
        print "Type \"INIT\" to continue - do you want to erase everything?\n";
        waitFor("INIT");
    } elsif ($ARGV[1] eq "wipe") {
        print "This will delete ALL session data in elastic search! (It does not delete the pcap files on disk or user info.)\n\n";
        print "Type \"WIPE\" to continue - do you want to wipe everything?\n";
        waitFor("WIPE");
    }
    print "Erasing\n";
    esDelete("/tags_v2", 1);
    esDelete("/tags", 1);
    esDelete("/sequence", 1);
    esDelete("/files_v3", 1);
    esDelete("/files_v2", 1);
    esDelete("/files_v1", 1);
    esDelete("/files", 1);
    esDelete("/stats", 1);
    esDelete("/dstats", 1);
    esDelete("/dstats_v1", 1);
    esDelete("/sessions*", 1);
    esDelete("/template_1", 1);
    if ($ARGV[1] eq "init") {
        esDelete("/users_v1", 1);
        esDelete("/users_v2", 1);
        esDelete("/users", 1);
    }
    esDelete("/tagger", 1);

    sleep(1);

    print "Creating\n";
    tagsCreate();
    sequenceCreate();
    filesCreate();
    statsCreate();
    dstatsCreate();
    sessionsUpdate();
    if ($ARGV[1] eq "init") {
        usersCreate();
    }
    print "Finished.  Have fun!\n";

Erasing previously captured pcap data

Done this only remains to remove pcap data from “raw” directory:

erasing moloch pcap data
1

~/moloch/db/raw# rm *

That’s all folks, enjoy your fresh baked moloch!

Comments