This blog is part of a series. Refer to the links below for the other posts in this series.
Part 1: Installing Ubuntu
Part 2: Installing Elastic Stack
Part 3: Install ElastiFlow
Part 4: Solution Maintenance (coming soon)
In parts 1 and 2 of this tutorial, we installed the Ubuntu server and Elastic Stack (ELK Stack). Now we are ready to install and configure ElastiFlow.
Before beginning, I recommend setting up at least one network device to start sending logs to the server. In my environment, I configured my pfSense firewall to send IPv4 flows using port 9995. It is important that you make note of the port you set up in your environment, as we will need to configure ElastiFlow to receive them as part of this tutorial.
The steps below are based on the directions found in ElastiFlow GitHub site. I’ve just expanded upon them and given you the command relevant to the Ubuntu and Elastic Stack 6.3 install we performed in parts 1 and 2. The instructions here are for ElastiFlow 3.x
Set JVM heap size.
It is recommended to set the JVM heap size to at least 2GB. If you are going to be doing DNS lookups then 4GB is recommended.
Open the jvm.options for Logstash to set the heap size
sudo nano /etc/logstash/jvm.options
Edit the Xms and Xmx sizes in the jvm.options configuration
Add and Update Required Logstash plugins
sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns
Copy the pipeline files to the Logstash configuration path
Create a temp folder to hold install files
Navigate to the temp folder you just created
Download Elastiflow install files
Install unzip, so you can extract the archive file you just downloaded
sudo apt-get install -y unzip
Unzip the Elastiflow files
Copy ElastiFlow configuration files to the Logstash directory
sudo cp -a elastiflow-master/logstash/elastiflow/. /etc/logstash/elastiflow/
Setup environment variable helper files
Copy the elastiflow.conf to systemd
sudo cp -a elastiflow-master/logstash.service.d/. /etc/systemd/system/logstash.service.d/
Add the ElastiFlow pipeline to pipelines.yml
Open the Logstash pipline configuration file for editing.
sudo nano /etc/logstash/pipelines.yml
Add the two line below to the bottom of the pipelines.yml file
- pipeline.id: elastiflow path.config: "/etc/logstash/elastiflow/conf.d/*.conf"
Open the elastiflow.conf file for editing.
sudo nano /etc/systemd/system/logstash.service.d/elastiflow.conf
The items you set here will be unique to your environment and setup. In my environment, I set the following:
ELASTIFLOW_NETFLOW_IPV4_HOST=<The Server’s IP Address>
Remember 9995 is the port I configured the network equipment to send flows on.
I also set ELASTIFLOW_RESOLVE_IP2HOST to true and set my DNS server in ELASTIFLOW_NAMESERVER so that the dashboards will attempt to resolve the DNS names instead of just displaying IP Address. There is a performance hit for this, but since it is just my lab network, it should not be a problem.
Ensure that the port for the incoming flows is open, on the firewall, so that Logstash is able to receive them.
sudo ufw allow from <IP Address> to any port 9995 proto tcp
Create logstash system startup script
Reload systemd manager configuration and start logstash
sudo systemctl daemon-reload sudo systemctl start logstash
Run the command below to check that logs are being received.
tail -f /var/log/logstash/logstash-plain.log
You should see log entries scrolling up the screen. Logstash can take some time to start so wait a few minutes after running the command. If after a little bit, it is just sitting there doing nothing, then either flows are not being sent or something is wrong with your configuration. If something is not configured correctly, you should see the error listed in the log. You can ignore any errors about there being nothing in the “/etc/logstash/conf.d/*.conf” folder. This is because we added the ElastiFlow to a second pipeline, so unless you previously setup anything on this server, than that folder should be empty.
Note: If using Netflow v9 or IPFIX you will likely see warning messages related to the flow templates not yet being received. They will disappear after templates are received from the network devices, which should happen every few minutes. Some devices can take a bit longer to send templates. Fortinet in particular send templates rather infrequently.
Hit Ctrl-C to exit from log tail
Assuming you are still in the flowtemp directory, run the command below to import the ElastiFlow indexes.
curl -X POST http://<Your Server's IP Address>:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @elastiflow-master/kibana/elastiflow.index_pattern.json
- On your local machine download the ElastiFlow dashboards. Right-click the link below choose save as https://github.com/robcowart/elastiflow/raw/master/kibana/elastiflow.dashboards.json
- Open your web browser and open the Kibana site.
- Navigate to Management > Advanced Settings
- Search for and set the recommended settings listed below. For details and additional information on what these are, refer to the ElastiFlow documentation.
doc_table:highlight false filters:pinnedByDefault true state:storeInSessionStorage true timepicker:quickRanges see link
- Navigate to Saved Objects and Import elastiflow.dashboards.json file you downloaded in step 1.
Once dashboard import completes you are done. You can now navigate to the Dashboard page in Kibana and start exploring the different visualizations. You can also check out the ElastiFlow Dashboard Documentation.