Download Shareware and Freeware Software for Windows, Linux, Macintosh, PDA

line Home  |  About Us  |  Link To Us  |  FAQ  |  Contact

Serving Software Downloads in 956 Categories, Downloaded 49.482.272 Times

jdresolve 0.6.1

  Date Added: July 27, 2010  |  Visits: 725

jdresolve

Report Broken Link
Printer Friendly Version


Product Homepage
Download (94 downloads)



jdresolve is a software that resolves IP addresses to hostnames. Any file format is supported, including those where the line does not begin with the IP address. One of the strongest features of the program is the support for recursion, which can drastically reduce the number of unresolved hosts by faking a hostname based on the network that the IP belongs to. DNS queries are sent in parallel, which means that you can decrease run time by increasing the number of simultaneous sockets used (given a fast enough machine and available bandwidth). By using the database support, performance can be increased even further, by using cached data from previous runs. HOW IT USED TO WORK jdresolve used the algorithms describe below up to version 0.2. The initial version of jdresolve tried to only speed up the name resolution by implementing numerous concurrent requests. I The first problem was: how to resolve the maximum possible number of IPs concurrently without reading the whole log file into memory (they can get quite _huge_)? I figured Id need a 2 pass approach, collecting all distinct host IPs that needing resolving in the first step, then resolving them efficiently inside a loop, and finally just replacing the resolved IPs on the second pass through the log file. This way we can garantee that the resolve queue will always be full with no need to weight that against how many lines of buffered log entries we would need to cache. The number of distinct IP addresses tend to be quite lower than the number of lines in the log file, and the IP part takes about only 1/20th of the log line, so we cant be using too much memory just by putting a few hundred or thousand small strings into a hash. After looking thru CPAN, I came across the excellent Net::DNS module and was more than happy to note that it already provide a subroutine and examples for background queries. Just add IO::Select to that and you have a full non-blocking aproach to multiple concurrent queries. You can even specify the timeouts to make the name resolving even more efficient. Having this much done, I was quite happy to have the fastest log resolving routine I have come accross. By setting the numbers of concurrent sockets and timeouts you could fine tune the beast to resolve names _very_ rapidly. But still there where about 25% of the IPs left unresolved... "This is not much help", I thought. I need to know _at least_ from what country these people are accessing from. After a few not very scientifical aproaches, I realized that by recurring thru the DNS classes (C, B and finally A) and checking for the host listed in the SOA record I could be pretty sure this was a father domain to the IP. The implementation goes like this: find out all distinct IP addresses, then determine which C, B and A classes contain these addresses. Make up a list from these queries and send them thru a resolver in chuncks of 32 (configurable via the command line). If a socket times out, leave that request unresolved. After running a big log file against the recursive aproach, I determined it didnt take much longer to resolve it at all. Full class domains tend to have decently configured DNS servers, and you get a lot of repeated classes when resolving your logs. The best was still to come: 0 unresolved IPs :) And since that I havent found an IP that cant be determined at least to its A class. HOW IT WORKS NOW The above algorithm works extremely well except for the case of very large logs (>100Mb). The hashes containing IPs and their parent A/B/C classes gets pretty huge doesnt fit in memory any more. So as of v0.3, we have a new 1 pass approach. We have a line cache that holds 10000 lines (configurable with -l, dont set it much lower). Using my test base it looks like each 10000 lines take about 4Mb of RAM during processing (thats the log lines themselves plus the hashes and arrays used for caching/processing). Each IP and class to be resolved has a count value, which is increased every time a line with that number is read, and decreased after we print out a resolved line with that reference value. Think of it as a "moving window" method, and that we do our own garbage collection. The process pauses if the first line in our line cache is still unresolved, we dont have any more sockets, or were waiting for socket data. We cant control the last two items, but to minimize the pauses do to yet unresolved lines, increase the -l value if you notice pauses during resolving. There should be enough lines cached so that even if we have timeouts on sockets we are still waiting for other socket data to come in, not just for 1 single socket to time out. Using this method the memory usage during executing is almost constant. So you can determine how much RAM you wish to use for resolving names and set your -l value and forget about it. Theres really no performance loss when compared to the <=v0.2 algorithm if you have a big enough line cache. HOW TO USE IT Example: jdresolve access_log > resolved.log If you simply run the script as you would with the Apache logresolve program, you get the same results, only much faster. But if you want really take advantage of jdresolve, you should at least turn on the -r option for recursive resolves. As of version 0.2, the -m option takes a mask as an argument. The valid substitutions are %i for the IP address and %c for the resolved class. So an IP like 1.2.3.4 with a mask of "%i.%c" (the default) would become something like "1.2.3.4.some.domain". A mask of "somewhere.in.%c" would turn it into "somewhere.in.some.domain". The -h switch shows you basic help information. The -v switch will display version information. Use -d 1 or -d 2 (more verbose) to debug the resolving process and get extra statistics. If you dont care for the default statistics, use -n to disable them. After some runs you may want to change your timeout value. The -t option accepts a new value in seconds. For even better performance, use the -s switch with a value greater then 32, but remember that many operating systems have a hard coded default for open files of 256 or 1024. Check your systems limit with "ulimit -a". New in v0.3 is the -l switch, which specified how many lines we will cache for resolving. The default is 10000, but can be vastly incremented without using too much RAM, as explained in "HOW IT WORKS". After you used jdresolve on the log file, you can check which ips where left unresolved by using the --unresolved option on the file that was generated. WHAT DOES RHOST DO? rhost is a quick script to take advantage of the new STDIN functionality of jdresolve. Many times you use the host command to resolve a single IP (like host 200.246.224.10). As with standard log resolvers, host doesnt do recursion. So rhost just calls jdresolve with the apropriate parameters to resolve that single IP number. The syntax is rhost . DATABASE SUPPORT As of version 0.5, jdresolve provides simple database support thru db (dbm, gdbm, sdbm, etc) files. You can use the --database switch to specify the db file and that will allow for fallback in case some DNS servers are down and also performance improvements since you can lower your timeout value without sacrificing resolved percentage. To use the database support, just supply a database name (i.e. hosts.db) using the --database option. If it does not yet exist, a new database with that name will be created. All resolved hosts and classes during a jdresolve run will be cached to the database. After you have some data in a db, you can use --dumpdb to look at it. With --mergedb to add new information to it (the format of the input file is the same as the one from a dump using --dumpdb, e.g. an ip/class followed by the hostname/classname, separated by white space) Ex: echo "0.0.0.0 testip" | jdresolve --database hosts.db --mergedb - ...adds and IP entry to the db Ex: echo "0.0.0 classname" | jdresolve --database hosts.db --mergedb - ...adds a class entry to the db Note: Since when recursing the resolved hostnames are stored to the database (even when resolved by recursion), you _may_ not want to use the same database for normal and recursed runs. That is because a cached host from a resolved run will show up as a "real" IP if you dont recurse and use the --dbfirst or --dbonly options, or just use the database and the lookup times out. Nothing too serious, but this detail may be important to some people..

Requirements: No special requirements
Platforms: Linux
Keyword: Database Database Support Dns File How Ip Addresses Ips It Jdresolve Log Log File Ram Resolved Resolving Using
Users rating: 0/10

License: Freeware Size: 51.2 KB
USER REVIEWS
More Reviews or Write Review


JDRESOLVE RELATED
Modules  -  Guests Other IP Addresses Remover 1.0.1
Clears the "Other IP addresses this user has posted from" part when viewing the IP of a Guest.
 
Networking Tools  -  Satellite 1.0.2
Satellite can track many remote machines with dynamic IP addresses in situations where public DNS services are inappropriate. Satellite can log and alert an admin immediately when a site comes online or needs attention. The Satellite archive...
53.25 KB  
DNS Tools  -  dnshistory 1.3
dnshistory project provide a means for storing a history of DNS and Name changes for the IP Addresses extracted from web log files. The major target being that multiple analyses of older log files do not require re-lookups of IP Address to...
112.64 KB  
Email  -  IlohaMail 0.8.14rc3 1.0
IlohaMail runs on a stock build of PHP, and does not require databases (although database support is available) or the IMAP library (it is powered by a custom IMAP/POP3 library). Admin Features - Easy to install (only requires default build of...
 
Misc. Networking Tools  -  TestIP 1.0
This utility is for resolving DNS names to IP addresses and for converting IP addresses to DNS names. It also shows aliases, if any. TestIP employs Winsock API and all info it returns in the status bar is what the particular winsock implementation...
18 KB  
Networking Tools  -  Arphound 1.3.1
Arphound project is a tools that listens to all traffic on an ethernet network interface, and reports IP/MAC address pair, as well as events such as IP changes, IP conflict, IP addresses with no RDNS, various ARP spoofing, and packets not using...
45.06 KB  
Networking  -  VNCTracker 2.8
VNCTracker provides database storage of IP Addresses for on-the-fly creation of .vnc connection files. IP's are learned either via a PHP POST form or tac/grepped from an email log. It currently uses Apache, PHP, MySQL, gnu-pop3d/courier-imap, tac,...
 
Utilities  -  PeerProtect 0.5
PeerProtect is an addon for Jays firewall that generates a file which contains all IP addresses from the RIAA and MPAA, etc. and will protect peer-to-peer programs from them. The Database can be copied from PEERGUARDIAN or IPPREFIX..
40.96 KB  
Libraries  -  IP::Country 2.23
IP::Country is a tool for fast lookup of country codes from IP addresses. Finding out the country of a client using only the IP address can be difficult. Looking up the domain name associated with that address can provide some help, but many IP...
163.84 KB  
Security Tools  -  PhishBlock 0.9.1.8
PhishBlock is a security program that detects and blocks Phishing, Pharming, C&C(Command and Control) Servers which are located in databases with URLs, DNS hostnames, and IP Addresses. This program detects and blocks Malware URLs, bad Hosts, and...
7.12 MB  
NEW DOWNLOADS IN NETWORK & INTERNET, NETWORKING TOOLS
Network & Internet  -  Free WiFi Hotspot 3.3.1
Free WiFi Hotspot is a super easy solution to turn your laptop or notebook into a portable Wi-Fi hotspot, wirelessly sharing your internet connections like DSL, Cable, Bluetooth, Mobile Broadband Card, Dial-Up, etc. through the built-in wireless...
1.04 MB  
Network & Internet  -  Easy Uploads 1.8
Easy uploads is a file storage media streaming application designed by Filestreamers that allows you to upload, store, and stream your files from their virtually unlimited file storage server. Easy Uploads can backup,share, and stream your files...
615.97 KB  
Network & Internet  -  PacketFence ZEN 3.1.0
PacketFence is a fully supported, trusted, Free and Open Source network access control (NAC) system. Boosting an impressive feature set including a captive-portal for registration and remediation, centralized wired and wireless management, 802.1X...
1024 MB  
Network & Internet  -  django-dbstorage 1.3
A Django file storage backend for files in the database.
10.24 KB  
Network & Internet  -  SQL Inject Me 0.4.5
SQL Inject Me is a Firefox extension used to test for SQL Injection vulnerabilities. The tool works by submitting your HTML forms and substituting the form value with strings that are representative of an SQL Injection attack.
133.12 KB  
Networking Tools  -  gvrpad 0.1
gvrpad is a daemon that makes GVRP announcements of all VLAN interfaces on a FreeBSD system. GVRP is the GARP VLAN Registration Protocol, defined in IEEE 802.1Q (VLANS); GARP is the Generic Attribute Registration Protocol, defined in 802.1D...
15.36 KB  
Networking Tools  -  Cheops 0.61
Cheops is an Open Source Network User Interface. It is designed to be the network equivalent of a swiss-army knife, unifying your network utilities. Cheops is for the network what a file manager is for your filesystem..
317.44 KB  
Networking Tools  -  ssh tunnel on demand 1.0
ssh tunnel on demand provides a script that creates an SSH tunnel on demand. ssh tunnel on demand is a script that makes it possible for a user to create an SSH tunnel to a server and connect to it without needing an account on the box or any...
13.31 KB  
Networking Tools  -  strongSwan 4.1.5
strongSwan is an OpenSource IPsec implementation for the Linux operating system. strongSwan is an OpenSource IPsec implementation for the Linux operating system. In order to have a stable IPsec platform to base our future extensions of the X.509...
1.7 MB  
Networking Tools  -  triggers 0.41
trigger is a lightweight, asynchronous notification mechanism to set off events in and across systems. The poor mans daily snapshot, glastree builds live backup trees, with branches for each day. Users directly browse the past to recover older...
14.34 KB