Elasticsearch / Logstash / Kibana (ELK)
This is another single-VM topology, containing the OSS (Apache2-licensed) version of the Elastic Stack, ready to install ElastiFlow for visualization of Netflow data.
These are heavyweight tools, and it's expected that you'll only run a single shared instance of them for the whole workshop. The students can all access the same web interface for exploring data.
This consists of a single virtual machine, elk.ws.nsrc.org (100.64.0.249). The VM has been configured with 8GB of RAM.
You will need the following files:
||the GNS3 project|
||the VM image with tools pre-installed (large download: ~1.2GB)|
||the cloud-init image which configures username/password and static IP|
|IP Address||DNS Name|
nsrc+ws(the standard student login). It's up to you whether you wish to keep this or change it.
The Kibana dashboard is available at http://kibana.ws.nsrc.org (no login). This is a virtualhost, to allow other tools to be added as further virtualhosts later if required.
ElastiFlow has a custom license which permits non-commercial use, but forbids redistribution, so it cannot be supplied in the pre-built VM.
Login to the VM, and then run the following script to download ElastiFlow and perform all the standard configuration:
This takes around 4 minutes to run, and then another 2-3 minutes for logstash to finish restarting before it can start processing flow data.
You will see the following warnings appear several times, but they can be ignored:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules to method sun.nio.ch.NativeThread.signal(long) WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release
Although Filebeat (OSS) is already installed, we provide a setup script for it as well:
This sets up the "system" module to read local logs from (
and also configures rsyslog to receive UDP port 514, so that you can use it
as a target for logs from other hosts.
ElastiFlow lists on IPv4 UDP port 2055 for Netflow traffic.
If you're running softflowd on the host, then you can change it to send
traffic to Elastiflow instead of nfdump/nfsen by changing
OPTIONS="-n 100.64.0.249:2055 -v 9 -t maxlife=5m"
ElastiFlow does not listen by default on IPv6 addresses, but it can be configured to do so.
Alternatively, you might want to configure one or more virtual Cisco routers
transit2.nren) to generate flow records including
NBAR application recognition, as Elastiflow will record this as
Here is a suggested configuration to apply:
flow record NBAR-V4 match application name match ipv4 tos match ipv4 protocol match ipv4 source address match ipv4 destination address match transport source-port match transport destination-port collect interface input collect interface output collect counter bytes long collect counter packets long ! flow record NBAR-V6 match application name match ipv6 traffic-class match ipv6 protocol match ipv4 source address match ipv4 destination address match transport source-port match transport destination-port collect interface input collect interface output collect counter bytes long collect counter packets long ! flow exporter EXPORTER-ELK description Export to elk destination 100.64.0.249 transport udp 2055 template data timeout 60 ! flow monitor FLOW-MONITOR-V4 exporter EXPORTER-ELK cache timeout active 300 record NBAR-V4 ! flow monitor FLOW-MONITOR-V6 exporter EXPORTER-ELK cache timeout active 300 record NBAR-V6 ! interface GigabitEthernet 0/0 ip flow monitor FLOW-MONITOR-V4 input ip flow monitor FLOW-MONITOR-V4 output ipv6 flow monitor FLOW-MONITOR-V6 input ipv6 flow monitor FLOW-MONITOR-V6 output
The mapping from NBAR IDs to application names is in
ELK is slow to start up, particularly Logstash: once it's running, you'll
want to keep it running. Make sure you select "Leave this project running
in the background when closing GNS3" under
File > Edit Project in the GNS3