This program package consists of scripts that automate the process of downloading, filtering, and encoding domain data from various lists. Here’s what it does:
1. Purpose of the Program: #
The primary goal of the program is to fetch and process domain lists from different categories (universal, fake news, gambling, pornography, social networks). Once downloaded, these domains are filtered and encoded in Base64 format.
2. Main Workflow Steps: #
Step 1: Downloading Files #
The script download_lists.py handles downloading domain lists from predefined URLs using the requests library. For each category, a separate .lst file is created.
URLs:
- Universal: General list.
- Fakenews: Domains associated with fake news.
- Gambling: Gambling-related sites.
- Porn: Pornographic content sites.
- Social: Social network domains.
Each downloaded list is saved locally as universal.lst, fakenews.lst, etc.
Step 2: Filtering Unique Subdomains #
The make_base_opt.sh script calls the prefilter_app program, which performs preliminary filtering on each downloaded file, removing duplicate or unnecessary entries. The filtered results are saved into temporary .tmp files (e.g., universal.tmp).
If an error occurs during this step, the script terminates with an error message.
Step 3: Base64 Encoding #
Next, the filtered data from the temporary files is encoded in Base64 format using the base64coder_app. The encoded results are saved into .blst files (e.g., universal.blst).
This ensures that all data is encoded in a format that is easy to transmit and store across various systems.
Step 4: Cleanup of Temporary Files #
After successful execution, the script removes temporary .tmp files and the original .lst files to free up space.
3. Purpose of the Components: #
make_base_opt.sh: The main control script. It orchestrates the sequence of processing steps.download_lists.py: Handles the downloading of source files.prefilter_app: Cleans up the data by removing unnecessary or duplicate subdomains.base64coder_app: Encodes the filtered data in Base64 format.
4. Typical Use Cases: #
- Domain Blocking: The encoded lists can be used to filter and block malicious sites in a system.
- Analysis: The prepared data can be loaded into IDS/IPS systems to monitor network traffic for suspicious domains.
- Network Security: Useful in firewalls, DNS servers, or proxy servers to block undesirable traffic categories (fake news, gambling, etc.).
5. Issues and Points of Attention: #
- The script stops on any error (e.g., failure in downloading or filtering).
- External programs
prefilter_appandbase64coder_appmust be precompiled and present in the same directory for the workflow to function correctly.
Conclusion: #
The program automates the cycle of downloading, filtering, and encoding domain lists from various categories, producing easy-to-use Base64-encoded lists for further use.
