[suricata]--file extract

# 0.参考
[1] [File Extraction](https://redmine.openinfosecfoundation.org/projects/suricata/wiki/File_Extraction)
[2] [suricata-5.0-file-extraction](https://suricata.readthedocs.io/en/suricata-5.0.0-beta1/file-extraction/file-extraction.html)
[3] [2011-file extraction in suricata](https://blog.inliniac.net/2011/11/29/file-extraction-in-suricata/)
[4] [2014-SMTP file extraction in Suricata](https://blog.inliniac.net/2014/11/11/smtp-file-extraction-in-suricata/)
[5] [suricata-5.0-file-extraction](https://suricata.readthedocs.io/en/suricata-5.0.0-beta1/rules/differences-from-snort.html?highlight=extraction#file-extraction)
[6] [file-keywords](https://suricata.readthedocs.io/en/suricata-5.0.0-beta1/rules/file-keywords.html)
[7] [Disabling Packet Checksum Validation](https://www.ibm.com/support/knowledgecenter/en/SS2MBL_9.0.2/PCAAdmin/PCA/Cfg/DisablingPacketChecksumValidation_28.html)

# 1.File Extract
Suricata has the ability to match on files from **FTP,HTTP and SMTP** streams and log **them** to **disk**(both uploads and downloads)

What files are actually extracted and stored to disk is **controlled by the rule language.**(rules and configuration)

Supported **protocols** are:

- HTTP
- SMTP
- FTP
- NFS
- SMB

<center>
![common-architecture](https://leanote.com/api/file/getImage?fileId=5d3bae93ab64413ce3003c79)
</center>

## 1.1 HTTP File Extract Architecture

- the file extraction code works on top of the HTTP Parser
 - HTTP Parser(largely a wrapper for libhtp) take care of dechunking and unzipping the request/response data

stream engine and reassembly engine and HTTP parser all affect the working of the file extraction

<center>
![architecture](https://leanote.com/api/file/getImage?fileId=5d3b99a8ab64413ce30038ba)
</center>

## 1.2 Settings 
```
stream.checksum_validation: no #(网卡控制checksum_checks,关掉命令sudo ethtool -K eth0 gro off lro off)
file-store.stream-depth:0 #(in suricata.yaml 0不限制，默认1 Megabyte)

libhtp.default-config.request-body-limit / libhtp.server-config.<config>.request-body-limit:0 #(in suricata.yaml,0 unlimited)

libhtp.default-config.response-body-limit / libhtp.server-config.<config>.response-body-limit:0 #(in suricata.yaml,0 unlimited)
```

## 1.3 Output
### 1.3.1 File-Store and Eve Fileinfo

two output modules for logging information about files extracted. eve里面的配置只记录metadata about the file,but not the actual file contents.

file-store stores the actual files to disk.

### 1.3.2 File-Store

> suricata.yaml

```
- file-store:
    enabled: yes      # set to yes to enable
    log-dir: files    # directory to store the files
    force-magic: no   # force logging magic on all stored files
    force-hash: [md5] # force logging of md5 checksums
    stream-depth: 1mb # reassemble 1mb into a stream, set to no to disable
    waldo: file.waldo # waldo file to store the file_id across runs
    max-open-files: 0 # how many files to keep open (O means none)
    write-meta: yes   # write a .meta file if set to yes
    include-pid: yes  # include the pid in filenames if set to yes.
```

# 1.4 file keywords

- filename
 - fileext
 - filemagic
 - filemd5
 - filesha1
 - filesha256
 - filesize

除保存文件外，额外记录的信息包括:

- timestamp
 - src/dst ip
 - protocol
 - src/dst port
 - HTTP URI
 - HTTP Host
 - HTTP Referer
 - filename
 - file magic
 - md5sum
 - size
 
> demo(metadata about the file store)

```
TIME: 01/27/2010-17:41:11.579196
PCAP PKT NUM: 2847035
SRC IP: 68.142.93.214
DST IP: 10.7.185.57
PROTO: 6
SRC PORT: 80
DST PORT: 56207
FILENAME: /msdownload/update/software/defu/2010/01/mpas-fe_7af9217bac55e4a6f71c989231e424a9e3d9055b.exe
MAGIC: PE32+ executable for MS Windows (GUI) Mono/.Net assembly
STATE: CLOSED
SIZE: 5204
```

# 1.5 Rules
## 1.5.1 HTTP examples
### 1.5.1.1 common
```
alter http any any -> any any (filename:"secret.xls";)#文件名+文件类型
alter http any any -> any any (fileext:"pdf")#文件类型
```
### 1.5.1.2 filemagic关键词
filemagic检索文件关键词，如果包含此关键词，就保存此文件
It  runs on the magic output of inspecting the start of a file.

```
alert http any any -> any any (filemagic:”PDF document”;)
alert http any any -> any any (filemagic:”PDF document, version 1.6″;)
```

>上传限定规则
This will alert on and store all files that are** uploaded** using a **POST** request that have a filename extension of **pdf**, but **the actual file is not pdf.**
```
alert http $EXTERNAL_NET any -> $HOME_NET any (msg:”pdf upload claimed, but not pdf”; flow:established,to_server; content:”POST”; http_method; fileext:”pdf”; filemagic:!”PDF document”; filestore; sid:1; rev:1;)
```

## 1.5.2 SMTP examples

include 4 parts:

- a **MIME** decoder(**MIME:Multipurpose Internet Mail Extensions**)
- updates to the SMTP parser to use the MIME decoder for extracting files
- SMTP JSON log, integrated with EVE
- SMTP message URL extraction and logging

> config(make sure that the **MIME decoder** is **enabled**)

```
app-layer:
  protocols:
    smtp:
      enabled: yes
      # Configure SMTP-MIME Decoder
      mime:
        # Decode MIME messages from SMTP transactions
        # (may be resource intensive)
        # This field supercedes all others because it turns the entire
        # process on or off
        decode-mime: yes

# Decode MIME entity bodies (ie. base64, quoted-printable, etc.)
        decode-base64: yes
        decode-quoted-printable: yes

# Maximum bytes per header data value stored in the data structure
        # (default is 2000)
        header-value-depth: 2000

# Extract URLs and save in state data structure
        extract-urls: yes
```

>logging(just add ‘smtp’ to the list of types in your EVE config)
```
# Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert:
            # payload: yes           # enable dumping payload in Base64
            # payload-printable: yes # enable dumping payload in printable (lossy) format
            # packet: yes            # enable dumping of packet (without stream segments)
            # http: yes              # enable dumping of http fields
        - http:
            extended: yes     # enable this for extended logging information
            # custom allows additional http fields to be included in eve-log
            # the example below adds three additional fields when uncommented
            #custom: [Accept-Encoding, Accept-Language, Authorization]
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: no   # force logging magic on all logged files
            force-md5: no     # force logging of md5 checksums
        #- drop
        - smtp
        - ssh
        # bi-directional flows
        #- flow
        # uni-directional flows
        #- newflow
```

信息安全从业人员^_^

导航

最近发表

友情链接