TK
Docs
Top Tags

Basic Bash monitoring tools

Basic Bash monitoring tools for network diagnostics and performance analysis

networkmonitoringbashdiagnosticstcpperformance

Overview

Network monitoring is essential for diagnosing connectivity issues, analyzing performance bottlenecks, and understanding traffic patterns. These bash-based tools provide low-level access to network statistics without requiring complex monitoring solutions.

Socket and Connection Monitoring

Understanding Socket States

Sockets represent endpoints for network communication. The Linux kernel maintains various states for TCP connections (ESTABLISHED, TIME_WAIT, CLOSE_WAIT, etc.) which are crucial for diagnosing connection issues.

Key socket states to monitor:

  • ESTABLISHED: Active connection with data transfer
  • TIME_WAIT: Connection closed but waiting to ensure remote TCP received acknowledgment
  • CLOSE_WAIT: Remote end has closed the connection, waiting for local application to close
  • SYN_SENT/SYN_RECV: Connection establishment in progress
  • FIN_WAIT: Connection termination in progress
bash
1# Check active connections and socket states
2ss -ant
3ss -s
4netstat -tapn | wc -l
5

6# Monitor network bandwidth
7iftop -i eth0
8

9# Capture packets for inspection
10tcpdump -i eth0 -n -c 100
11

12# View kernel logs for network errors
13dmesg | grep TCP
14

15# View connection and error metrics
16cat /proc/net/snmp
17cat /proc/net/netstat

Tool Explanations

ss (Socket Statistics)

The ss command is the modern replacement for netstat, providing faster socket statistics from kernel space.

  • ss -ant: Shows all (-a) numeric (-n) TCP (-t) sockets, avoiding DNS resolution for faster output
  • ss -s: Displays summary statistics including total sockets, TCP connections in various states, UDP sockets, and raw sockets

netstat

Legacy but still widely used tool for network statistics. The pipe to wc -l counts total active connections, useful for detecting connection leaks or DDoS attacks.

iftop (Interface TOP)

Real-time bandwidth monitoring tool that displays current bandwidth usage on a network interface. Essential for:

  • Identifying bandwidth-heavy connections
  • Detecting unexpected traffic patterns
  • Monitoring upload/download rates per connection
  • The -i eth0 flag specifies the interface to monitor (replace with your interface name like enp0s3, wlan0, etc.)

tcpdump

Powerful packet analyzer that captures raw network traffic at the data link layer. The command shown:

  • -i eth0: Captures on ethernet interface eth0
  • -n: Disables name resolution for faster capture
  • -c 100: Limits capture to 100 packets

Use cases: Protocol analysis, debugging application-level issues, security investigation, verifying packet delivery

dmesg (Diagnostic Message)

Kernel ring buffer contains low-level system messages. Filtering for TCP messages reveals:

  • Connection timeouts
  • TCP retransmission issues
  • Socket buffer overflows
  • Network driver errors
  • Kernel-level connection problems

/proc/net/ Statistics

These pseudo-files expose kernel network stack statistics:

/proc/net/snmp: SNMP MIB-II statistics including:

  • IP packet counts (received, delivered, forwarded, discarded)
  • ICMP message types and counts
  • TCP segments (sent, received, retransmitted)
  • UDP datagram statistics

/proc/net/netstat: Extended TCP/IP statistics including:

  • SYN cookies usage
  • TCP Fast Open statistics
  • Packet loss and recovery metrics
  • Memory pressure indicators
  • TCP congestion control events

Best Practices

  1. Baseline your metrics: Run these tools during normal operation to establish baseline values
  2. Combine tools: Use ss for quick state checks, tcpdump for deep packet inspection, and iftop for bandwidth analysis
  3. Time correlation: When investigating issues, correlate timestamps across different tools
  4. Interface selection: Always verify you're monitoring the correct network interface
  5. Privilege requirements: Most tools require root/sudo for full functionality

Common Diagnostic Scenarios

  • High TIME_WAIT states: May indicate application not reusing connections properly
  • Growing CLOSE_WAIT: Application not closing sockets after remote disconnect
  • Retransmission spikes in /proc/net/netstat: Network quality issues or congestion
  • Bandwidth saturation in iftop: Need for QoS or capacity planning

Advanced Connection Analysis

Monitoring Specific Applications and Ports

Track network activity for specific applications using process IDs and port numbers:

bash
1# Find connections by process name
2ss -tnp | grep nginx
3lsof -i -n | grep ESTABLISHED
4

5# Monitor specific port
6ss -tnp state established '( dport = :80 or sport = :80 )'
7netstat -an | grep :443 | grep ESTABLISHED | wc -l
8

9# Check which process is listening on a port
10lsof -i :8080
11ss -tlnp | grep :3306
12

13# List all listening ports with process info
14ss -tulpn
15netstat -tulpn

Continuous Monitoring and Alerting

Set up continuous monitoring with thresholds for anomaly detection:

bash
1# Watch connection count in real-time (refreshes every 2 seconds)
2watch -n 2 'ss -s'
3

4# Monitor connections exceeding threshold
5while true; do
6  COUNT=$(ss -tan state established | wc -l)
7  if [ $COUNT -gt 1000 ]; then
8    echo "$(date): High connection count: $COUNT" >> /var/log/conn_monitor.log
9  fi
10  sleep 10
11done
12

13# Track TIME_WAIT accumulation
14watch -n 1 'ss -tan state time-wait | wc -l'
15

16# Monitor packet drops on interface
17watch -n 1 'ethtool -S eth0 | grep -i drop'

Traffic Analysis and Packet Inspection

Protocol-Specific Monitoring

Deep dive into specific protocols for troubleshooting:

bash
1# Capture HTTP traffic with headers
2tcpdump -i eth0 -A -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
3

4# Monitor DNS queries and responses
5tcpdump -i eth0 -n port 53
6

7# Capture HTTPS handshake (TLS/SSL)
8tcpdump -i eth0 -n port 443 -w https_capture.pcap
9

10# Monitor ICMP (ping) packets
11tcpdump -i eth0 icmp
12

13# Filter by IP address
14tcpdump -i eth0 host 192.168.1.100
15

16# Capture traffic between two hosts
17tcpdump -i eth0 'host 192.168.1.10 and host 192.168.1.20'

Network Performance Metrics

Measure latency, throughput, and packet loss:

bash
1# Check network interface statistics
2ip -s link show eth0
3cat /sys/class/net/eth0/statistics/rx_bytes
4cat /sys/class/net/eth0/statistics/tx_bytes
5

6# Monitor packet errors and drops
7netstat -i
8ip -s -s link show eth0
9

10# Continuous bandwidth measurement
11nload eth0
12bmon -p eth0
13

14# Network throughput testing
15iperf3 -s  # On server
16iperf3 -c server_ip -t 30  # On client
17

18# Measure latency continuously
19ping -i 0.2 -c 100 8.8.8.8 | tail -n 3

System-Level Network Diagnostics

Kernel and Driver Information

Check low-level network configuration and performance:

bash
1# View network device driver info
2ethtool eth0
3ethtool -i eth0  # Driver details
4ethtool -g eth0  # Ring buffer parameters
5ethtool -k eth0  # Offload settings
6

7# Check interrupt handling
8cat /proc/interrupts | grep eth0
9mpstat -I ALL 1 5
10

11# Monitor network buffer usage
12sysctl net.core.rmem_max
13sysctl net.core.wmem_max
14sysctl net.ipv4.tcp_rmem
15sysctl net.ipv4.tcp_wmem
16

17# Check routing table and ARP cache
18ip route show
19ip neigh show
20arp -an

Connection State Monitoring Scripts

Automated analysis of connection patterns:

bash
1# Count connections by state
2ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn
3

4# Top IPs by connection count
5ss -tan state established | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -10
6

7# Monitor new connections per second
8for i in {1..60}; do
9  echo "$(date +%T): $(ss -tan state syn-sent,syn-recv | wc -l) new connections"
10  sleep 1
11done
12

13# Detect potential port scans
14netstat -tan | grep SYN_RECV | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn
15

16# Connection duration analysis
17ss -tno state established | grep timer | wc -l

Real-Time Network Event Monitoring

Using nethogs and vnstat

Track bandwidth by process and historical data:

bash
1# Real-time per-process bandwidth monitoring
2sudo nethogs eth0
3

4# Historical network statistics
5vnstat -i eth0
6vnstat -i eth0 -h  # Hourly stats
7vnstat -i eth0 -d  # Daily stats
8vnstat -i eth0 -m  # Monthly stats
9

10# Live traffic rates
11vnstat -i eth0 -l

Network Flow Analysis

Monitor network flows for traffic pattern analysis:

bash
1# Count active flows
2ss -tan state established | awk '{print $4" "$5}' | sort | uniq -c | sort -rn | head -20
3

4# Track connection establishment rate
5sar -n TCP 1 10
6

7# Monitor network device queues
8tc -s qdisc show dev eth0
9

10# Check conntrack table (for NAT/firewall)
11conntrack -L | wc -l
12cat /proc/sys/net/netfilter/nf_conntrack_count
13cat /proc/sys/net/netfilter/nf_conntrack_max

Performance Troubleshooting Commands

Identifying Bottlenecks

Diagnose network stack performance issues:

bash
1# Check for TCP retransmissions
2ss -ti | grep -i retrans
3

4# Monitor socket receive queue depth
5ss -tnm | grep -i rcv
6

7# Check for window scaling issues
8cat /proc/net/netstat | grep TcpExt | grep -i window
9

10# View TCP congestion control algorithm
11sysctl net.ipv4.tcp_congestion_control
12ss -i | grep -i cubic
13

14# Check for SYN flood protection
15netstat -s | grep -i listen
16cat /proc/sys/net/ipv4/tcp_max_syn_backlog
17

18# Monitor socket memory pressure
19cat /proc/net/sockstat
20cat /proc/net/sockstat6