CanadianInstituteForCybersecurity · ekky-kharismadhany · Feb 5, 2020 · Feb 5, 2020 · Feb 5, 2020 · Feb 5, 2020
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,5 @@
+.gradle
+.idea
+build
+logs
+data
diff --git a/LICENSE.txt b/LICENSE.txt
@@ -1,32 +1,18 @@
-Copyright (c) 2016 Canadian Institute for Cybersecurity (CIC)
+# Copyright (c) 2016 
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (CICFlowMeter), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (CICFlowMeter), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
-For citation in your works and also understanding CICFlowMeter (formerly ISCXFlowMeter) completely, you can find below published papers:
+# For citation in your works and also understanding CICFlowMeter (formerly ISCXFlowMeter) completely, you can find below published papers:
 
 Arash Habibi Lashkari, Gerard Draper-Gil, Mohammad Saiful Islam Mamun and Ali A. Ghorbani, "Characterization of Tor Traffic Using Time Based Features", In the proceeding of the 3rd International Conference on Information System Security and Privacy, SCITEPRESS, Porto, Portugal, 2017
 
 Gerard Drapper Gil, Arash Habibi Lashkari, Mohammad Mamun, Ali A. Ghorbani, "Characterization of Encrypted and VPN Traffic Using Time-Related Features", In Proceedings of the 2nd International Conference on Information Systems Security and Privacy(ICISSP 2016) , pages 407-414, Rome , Italy
 
-Contributors:
+# Contributors:
 Arash Habibi Lashkari: Researcher and Developer (Founder)
-
 Gerard Drapper: Researcher and Developer (Co-funder)
-
 muhammad saiful islam: Researcher and Developer (Co-funder)
diff --git a/README.md b/README.md
@@ -13,7 +13,7 @@ open a Terminal in the IDE
 ```
 //linux:
 $ sudo bash
-$ gradle execute
+$ ./gradlew execute
 
 //windows:
 $ gradlew execute
@@ -35,7 +35,7 @@ Run eclipse with sudo
 open a Terminal in the IDE
 ```
 //linux:
-$ gradle distZip
+$ ./gradlew distZip
 //window
 $ gradlew distZip
 ```

diff --git a/ReadMe.txt b/ReadMe.txt
@@ -1,81 +1,15 @@
-# Intro
 The CICFlowMeter is an open source tool that generates Biflows from pcap files, and extracts features from these flows.
 
 CICFlowMeter is a network traffic flow generator available from here . It can be used to generate bidirectional flows, where the first packet determines the forward (source to destination) and backward (destination to source) directions, hence the statistical time-related features can be calculated separately in the forward and backward directions. Additional functionalities include, selecting features from the list of existing features, adding new features, and controlling the duration of flow timeout.
 
 NOTE: TCP flows are usually terminated upon connection teardown (by FIN packet) while UDP flows are terminated by a flow timeout. The flow timeout value can be assigned arbitrarily by the individual scheme e.g., 600 seconds for both TCP and UDP.
 
-For citation in your works and also understanding CICFlowMeter (formerly ISCXFlowMeter) completely, you can find below published paper:
-Gerard Drapper Gil, Arash Habibi Lashkari, Mohammad Mamun, Ali A. Ghorbani, "Characterization of Encrypted and VPN Traffic Using Time-Related Features", In Proceedings of the 2nd International Conference on Information Systems Security and Privacy(ICISSP 2016) , pages 407-414, Rome , Italy
-
-
-----------------------------------------
-
-# Installation and executing:
-
-Extract CICFlowMeterV3.zip
-
-___Note: The only prerequisite is that "libpcap" library or WinPcap on windows systems, be pre-installed___
-
-
-For Linux
-
-> $ sudo apt-get install libpcap-dev
-
-
-For windows
-> download [winpcap](<https://www.winpcap.org/install/default.htm>)
-
-## executing
-Go to the extracted directory,enter the 'bin' folder
-
-### linux
-Open a terminal and run this command
-```
-//For GUI:
-sudo ./CICFlowMeter
-
-//For Command line:
-./cfm "inputFolder" "outputFolder"
-```
-### windows
-Lanunch the Comand Prompt and run this command
-```
-//for GUI:
-CICFlowMeter.bat
-
-//for Commond line:
-cfm.bat "inputFolder" "outputFolder"
-```
-
-## Get started
-for offline
-```
-1.Select the folder that include your PCAP files
-2.Select the folder that you would like to save you CSV files
-3.Click OK button
-```
-
-for realtime
-```
-1 CLick Load button to find the list of network interfaces
-2 Select the interface you would like to monitor
-3 Click start button and wait for a while
-4 Click stop button to stop the process and save the csv in same applcation folder/data/daily
-```
-
---------------------------------------------------------------
-
-Contact us at [email protected] if there are any problems.
-
-
 For citation in your works and also understanding CICFlowMeter (formerly ISCXFlowMeter) completely, you can find below published papers:
 
 Arash Habibi Lashkari, Gerard Draper-Gil, Mohammad Saiful Islam Mamun and Ali A. Ghorbani, "Characterization of Tor Traffic Using Time Based Features", In the proceeding of the 3rd International Conference on Information System Security and Privacy, SCITEPRESS, Porto, Portugal, 2017
 
 Gerard Drapper Gil, Arash Habibi Lashkari, Mohammad Mamun, Ali A. Ghorbani, "Characterization of Encrypted and VPN Traffic Using Time-Related Features", In Proceedings of the 2nd International Conference on Information Systems Security and Privacy(ICISSP 2016) , pages 407-414, Rome , Italy
 
-
 --------------------------------------------------------------
 List of extracted features and descriptions:
 Feature Name				Description
@@ -92,8 +26,8 @@ Bwd Packet Length Min		Minimum size of packet in backward direction
 Bwd Packet Length Max		Maximum size of packet in backward direction
 Bwd Packet Length Mean		Mean size of packet in backward direction
 Bwd Packet Length Std		Standard deviation size of packet in backward direction
-Flow Byte/s			Number of flow packets per second
-Flow Packets/s			Number of flow bytes per second 
+Flow Bytes/s			Number of flow bytes per second
+Flow Packets/s			Number of flow packets per second 
 Flow IAT Mean			Mean time between two packets sent in the flow
 Flow IAT Std			Standard deviation time between two packets sent in the flow
 Flow IAT Max			Maximum time between two packets sent in the flow
@@ -108,16 +42,16 @@ Bwd IAT Max			Maximum time between two packets sent in the backward direction
 Bwd IAT Mean			Mean time between two packets sent in the backward direction
 Bwd IAT Std			Standard deviation time between two packets sent in the backward direction
 Bwd IAT Total			Total time between two packets sent in the backward direction
-Fwd PSH flag			Number of times the PSH flag was set in packets travelling in the forward direction (0 for UDP)
-Bwd PSH Flag			Number of times the PSH flag was set in packets travelling in the backward direction (0 for UDP)
-Fwd URG Flag			Number of times the URG flag was set in packets travelling in the forward direction (0 for UDP)
-Bwd URG Flag			Number of times the URG flag was set in packets travelling in the backward direction (0 for UDP)
+Fwd PSH flags			Number of times the PSH flag was set in packets travelling in the forward direction (0 for UDP)
+Bwd PSH Flags			Number of times the PSH flag was set in packets travelling in the backward direction (0 for UDP)
+Fwd URG Flags			Number of times the URG flag was set in packets travelling in the forward direction (0 for UDP)
+Bwd URG Flags			Number of times the URG flag was set in packets travelling in the backward direction (0 for UDP)
 Fwd Header Length		Total bytes used for headers in the forward direction
 Bwd Header Length		Total bytes used for headers in the backward direction
 FWD Packets/s			Number of forward packets per second
 Bwd Packets/s			Number of backward packets per second
-Min Packet Length 		Minimum length of a packet
-Max Packet Length 		Maximum length of a packet
+Packet Length Min 		Minimum length of a packet
+Packet Length Max		Maximum length of a packet
 Packet Length Mean 		Mean length of a packet
 Packet Length Std		Standard deviation length of a packet
 Packet Length Variance  	Variance length of a packet
@@ -127,27 +61,26 @@ RST Flag Count 			Number of packets with RST
 PSH Flag Count 			Number of packets with PUSH
 ACK Flag Count 			Number of packets with ACK
 URG Flag Count 			Number of packets with URG
-CWR Flag Count 			Number of packets with CWE
+CWR Flag Count 			Number of packets with CWR
 ECE Flag Count 			Number of packets with ECE
 down/Up Ratio			Download and upload ratio
 Average Packet Size 		Average size of packet
-Avg Fwd Segment Size 		Average size observed in the forward direction
-AVG Bwd Segment Size 		Average number of bytes bulk rate in the backward direction
-Fwd Header Length		Length of the forward packet header
-Fwd Avg Bytes/Bulk		Average number of bytes bulk rate in the forward direction
-Fwd AVG Packet/Bulk 		Average number of packets bulk rate in the forward direction
-Fwd AVG Bulk Rate 		Average number of bulk rate in the forward direction
-Bwd Avg Bytes/Bulk		Average number of bytes bulk rate in the backward direction
-Bwd AVG Packet/Bulk 		Average number of packets bulk rate in the backward direction
-Bwd AVG Bulk Rate 		Average number of bulk rate in the backward direction
+Fwd Segment Size Avg 		Average size observed in the forward direction
+Bwd Segment Size Avg 		Average size observed in the backward direction
+Fwd Bytes/Bulk Avg		Average number of bytes bulk rate in the forward direction
+Fwd Packet/Bulk Avg		Average number of packets bulk rate in the forward direction
+Fwd Bulk Rate Avg 		Average number of bulk rate in the forward direction
+Bwd Bytes/Bulk Avg		Average number of bytes bulk rate in the backward direction
+Bwd Packet/Bulk Avg 		Average number of packets bulk rate in the backward direction
+Bwd Bulk Rate Avg		Average number of bulk rate in the backward direction
 Subflow Fwd Packets		The average number of packets in a sub flow in the forward direction
 Subflow Fwd Bytes		The average number of bytes in a sub flow in the forward direction
 Subflow Bwd Packets		The average number of packets in a sub flow in the backward direction
 Subflow Bwd Bytes		The average number of bytes in a sub flow in the backward direction
-Init_Win_bytes_forward		The total number of bytes sent in initial window in the forward direction
-Init_Win_bytes_backward		The total number of bytes sent in initial window in the backward direction
-Act_data_pkt_forward		Count of packets with at least 1 byte of TCP data payload in the forward direction
-min_seg_size_forward		Minimum segment size observed in the forward direction
+Fwd Init Win bytes		The total number of bytes sent in initial window in the forward direction
+Bwd Init Win bytes		The total number of bytes sent in initial window in the backward direction
+Fwd Act Data Pkts		Count of packets with at least 1 byte of TCP data payload in the forward direction
+Fwd Seg Size Min		Minimum segment size observed in the forward direction
 Active Min			Minimum time a flow was active before becoming idle
 Active Mean			Mean time a flow was active before becoming idle
 Active Max			Maximum time a flow was active before becoming idle
@@ -157,3 +90,5 @@ Idle Mean			Mean time a flow was idle before becoming active
 Idle Max			Maximum time a flow was idle before becoming active
 Idle Std			Standard deviation time a flow was idle before becoming active
 
+--------------------------------------------------------------------------------------
+
diff --git a/build.gradle b/build.gradle
@@ -29,6 +29,8 @@ dependencies {
     compile group: 'com.google.guava', name: 'guava', version: '23.6-jre'
     // https://mvnrepository.com/artifact/org.apache.tika/tika-core
     compile group: 'org.apache.tika', name: 'tika-core', version: '1.17'
+    compile group: 'com.google.code.gson', name: 'gson', version: '2.8.9'
+    compile group: 'org.apache.kafka', name: 'kafka-clients', version: '3.1.0'
 
 }
 sourceSets {
@@ -94,8 +96,7 @@ task fatJar(type: Jar) {
 
 task execute(type: JavaExec) {
     println 'type JavaExec'
-
-
+    standardInput = System.in
     main = "cic.cs.unb.ca.ifm.App" //main class
     classpath = sourceSets.main.runtimeClasspath
     String osName = System.getProperty('os.name').toLowerCase()

diff --git a/src/main/java/cic/cs/unb/ca/ifm/App.java b/src/main/java/cic/cs/unb/ca/ifm/App.java
@@ -6,6 +6,7 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import java.awt.*;
+import java.util.Scanner;
 
 public class App {
 	public static final Logger logger = LoggerFactory.getLogger(App.class);
@@ -18,19 +19,6 @@ public static void init() {
 	 * Launch the application.
 	 */
 	public static void main(String[] args) {
-		/*try {
-			//UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
-			UIManager.setLookAndFeel("javax.swing.plaf.nimbus.NimbusLookAndFeel");
-		} catch (ClassNotFoundException e1) {
-			e1.printStackTrace();
-		} catch (InstantiationException e1) {
-			e1.printStackTrace();
-		} catch (IllegalAccessException e1) {
-			e1.printStackTrace();
-		} catch (UnsupportedLookAndFeelException e1) {
-			e1.printStackTrace();
-		}*/
-
 		EventQueue.invokeLater(() -> {
             try {
                 init();

diff --git a/src/main/java/cic/cs/unb/ca/ifm/CICFlowMeter.java b/src/main/java/cic/cs/unb/ca/ifm/CICFlowMeter.java
@@ -105,10 +105,5 @@ public boolean accept(File dir, String name) {
 		}
 		logger.info("\n\n----------------------------------------------------------------------------\n TOTAL FLOWS GENERATED: {}",totalFlows);
 		logger.info("----------------------------------------------------------------------------\n");
-//		try {
-//			Files.write(Paths.get("src/main/resources/executionLog.log"),flowGen.dumpFlows().getBytes());
-//		} catch (IOException e) {			
-//			e.printStackTrace();
-//		}
 	}
 }
-Original file line number
+Diff line change
@@ -0,0 +1,5 @@
+    .gradle
+    .idea
+    build
+    logs
+    data