Blocking ads and trackers using HOSTS

If you’ve stumbled across this post, you’re probably familiar with adblocking extensions such as Adblock and uBlock(seriously recommend the latter for a handful of reasons) and most likely you’re in need of a solution to take back your network and system resources as well as a need for less clutter and more privacy in your daily web ventures, however, this method for blocking ads at the browser level only tends to be quite inefficient and fairly limited. Wouldn’t it be cool to also have ads and trackers blocked at the system level, including but not limited to applications like Skype, uTorrent, IE(seriously?) and other browsers or the many shareware/freeware apps that track your usage via mechanisms like Google Analytics(some use exactly that for tracking).

The solution is fairly simple, we’re going to use a simple hostname based block list to map undesirable domain names to either 0.0.0.0 or 127.0.0.1. In my testing on OS X, I found that 0.0.0.0 works best, that might not be the case on different operating systems. The blocking is done via the ages old hosts(5) unix file, but still very useful mechanism for easy static ip-name mappings at the host level.

The current block list that I use is hosted at hosts.neocities.org. I’m not affiliated with that site and don’t know who is providing it, that being said I use git to track and review changes between updates. The list is quite exhaustive, combining lists from several other sources cited in the header. I’d like to see a couple more lists combined like that from several other places(mainly the ones from uBlock would be useful), but you can then add extra lists by modifying the script fairly easily.

Now the script itself, is hosted on Github. Please read the entire script and what I’ve written bellow before running the script on your system.

Before you go on and use the script on your OS X, I really encourage to start using git in your /etc/ directory. The script won’t even work without a git repo in /etc/, unless you know what you’re doing and you’re going to modify it to bypass that. Having a git repo in your etc directory gives you revisioning, rollback, beta-testing, review and scrutiny abilities to whatever you’re doing to your etc. I do this on my workstations, laptops and servers that I manage. The added git overhead on your daily etc routines is insignificant when compared to the benefits you get when you most need them.

The script is smart enough to not break your current system. What it does as part of the first time run initialization is copy your current /etc/hosts to /etc/hosts.d/hosts.1.head. All your existing localhost rules and custom rules will be maintained there. The adblocking rules will go into /etc/hosts.d/hosts.3.adblock. You can add custom mapping rules(for staging servers, local network mappings) to  hosts.2.custom.

Then each time the script updates it will do the following:

  1. Update hosts.3.adblock with the latest rules from upstream;
  2. Concatenate the rules in /etc/hosts.d in the numeric order to your /etc/hosts;
  3. Show you a git diff of the changes and the option to commit those changes or deny to review, undo or commit yourself using git;

The script also has some pfsense blocking rules from www.emergingthreats.net and some custom ip blocking enabled in /etc/pf.rules/ip-block.pf. This is disabled by default, you can enable it by setting the PFSENSE var to “true” or passing -f as argument. If you know of some other worthy and fresh ad/malware ip lists let me know.

Although my script is OS X only, it’s fairly easy to port it to any other UNIX system(I welcome patches to the main script via Github), having such a solution for the Windows platform would be cool too. Maybe someone reading this can weigh in with their solution or insight? Would it work fair enough, is cygwin the only way for automating this? Nonetheless, stay tuned, since I have a similar router solution(AsusWRT, DD-WRT) coming up soon, that steps up the game a notch and provides blocking for your entire network, though it surely doesn’t deprecate this host level solution (on a laptop for e.g. that is frequently switching networks).

Pros for this setup:

  1. Easy setup and update (when compared to a firewall or a custom dns);
  2. Cross-platform and cross-application solution;
  3. Faster and less intrusive(also no https mitm) than proxy solutions(such as Privoxy);
  4. Easy to temporarily disable: just cp /etc/hosts.d/hosts.1.head /etc/hosts and to restore git checkout /etc/hosts;

Caveats:

  1. On some operating systems hosts files with tens of thousands of rules might slow name resolution up to a certain degree. In my usage with over 50000 rules, OS X and Linux is quite fine in that regard. If you find that such is your case, maybe using a dns server or firewall rules is better for you;
  2. Some blank spaces, containers, divs or unresolved error messages will take the place of the ads themselves in sites and apps that don’t handle failure very well. You can get rid of the browser related blanks at least by using uBlock extension with just the cosmetic rules enabled(in the extension Settings);
  3. Related to the previous one, you might experience some failures in certain web related functionality(fairly limited though). Most of them will be social related or news sites that use ad nag pages before they redirect you to the article content itself. Personally I don’t care about them and as soon as I hit such a road block I close it and move on. The benefit of more resources and network bandwidth for my system as well as the increased privacy and less clutter in general, totally trumps any minor drawback like this;
  4. The script relies on the links(1)(or elinks) tool to parse the html page at hosts.neocities.org and extract only the text. On OS X I use homebrew to install additional tools that I need. If you have a better solid solution that relies only on coreutils or other commonly installed shell utilities let me know;

 

 

Tagged with:
 

Alternative to the finger Unix command

Every once in a while I stumble on this question on yet too many forums and Q&A sites:

Is there an alternative to the finger command?

The simple answer is yes! There is an official replacement for the original finger command and it’s part of the gnu coreutils package, it is called pinky(1) and it’s available on all systems that use the gnu coreutils.

The long answer is, no, there isn’t really a real replacement to the original remote enabled finger protocol along with the fingerd daemon and finger client. While it did make sense and has seen its fair share of usage in the early days of the Internet,  it has been considered obsolete and a security issue for way over a decade now, thus all of the modern Linux distributions and Unix flavors don’t install the service nor the client by default anymore, some don’t even include them in their repositories at all.

So now, most of the replacement tools and commands that one can use instead of finger are going to act on the local machine only and provide logged in user info only for the current host. One such tool is pinky of course and it’s output similar to:

$ pinky
Login Name TTY Idle When Where
shinnok Shinnok pts/0 2013-09-18 07:32 127.0.0.1
andy        Andy pts/1 2013-09-18 07:32 127.0.0.2

As you can see the output is pretty similar to the who(1) command(it was actually ported from who.c), nothing special about it. The other traditional alternatives to the finger command include w(1) and users(1).

Tagged with:
 

It’s about time to call the end on the current poll with 412 votes, Which OS do you currently use the most? , and draw the results:

The best pentesting distribution is?

  • Backtrack (62%, 252 Votes)
  • Other (18%, 74 Votes)
  • nUbuntu (9%, 36 Votes)
  • Network Security Toolkit (2%, 9 Votes)
  • General Knoppix (2%, 8 Votes)
  • Pentoo (2%, 7 Votes)
  • Samurai (2%, 7 Votes)
  • Helix (1%, 4 Votes)
  • STD (1%, 4 Votes)
  • INSERT (1%, 3 Votes)
  • Operator (1%, 3 Votes)

Total Voters: 412

Loading ... Loading ...

The new poll is:

The most useful skills in a pentester's arsenal?

  • Out of the box, unconventional thinking (hacker mindset) (22%, 24 Votes)
  • Constant curiosity (also hacker mindset) (19%, 21 Votes)
  • Social engineering (15%, 16 Votes)
  • Programming and scripting (13%, 14 Votes)
  • Exploit development (8%, 9 Votes)
  • Vulnerability discovery and Fuzzing (7%, 8 Votes)
  • Tool knowledge (6%, 7 Votes)
  • Domain specific knowledge (like Networking, Operating Systems or Scada apps) (6%, 7 Votes)
  • Forensics (2%, 2 Votes)

Total Voters: 70

Loading ... Loading ...

This new poll is multi-choice, with a maximum of 3 selections allowed. So go ahead, and pick the three most useful skills you think a pentester should posses.

The “gravy sucking pig dog” in BSD

There’s been a while since BSD systems have this easter egg in the shutdown command source, “Die you gravy sucking pig dog” embedded into the name of the function the performs the actual halt or reboot.

FreeBSD has it(line 96):

void die_you_gravy_sucking_pig_dog(void);

OpenBSD has it(line 93):

void __dead die_you_gravy_sucking_pig_dog(void);

NetBSD has it(line 100):

static void die_you_gravy_sucking_pig_dog(void) __dead;

Even Apple has it too, but they #ifdef-ed it to another name since they don’t condone that kind of shit:

#ifdef __APPLE__
void log_and_exec_reboot_or_halt(void);
#else
void die_you_gravy_sucking_pig_dog(void);
#endif

This kind of source code profanity is really common in the open source world ,thus no news here. 🙂

I wonder for how long has this gravy sucking pig dog been wondering the BSD sources, maybe it can be traced back to Berkley BSD? If you happen to know, please feel free to share with us in a comment to this post.

Oh, almost forgot, and Linux is not better, for eg. here is a graph of a small selection of swear words in the Linux kernel or if you wan’t to see a more entertaining piece, then grab a bag of popcorn and read through file:///Sebastian/Droge/please/choke/on/a/bucket/of/cocks(that’s an http url ofc) bug on Debian. Enjoy.

Tagged with:
 

Command-line Fu

cmdfu

Command-Line-Fu is the place to record those command-line gems that you return to again and again.

Delete that bloated snippets file you’ve been using and share your personal repository with the world. That way others can gain from your CLI wisdom and you from theirs too. All commands can be commented on and discussed – digg-esque voting is also encouraged so the best float to the top.

A very good compilation of command line tricks. Ninja Style!

Tagged with:
 

Debian GNU/Linux 5.0 released

debian_lenny

The Debian Project is pleased to announce the official release of Debian GNU/Linux version 5.0 (codenamed Lenny) after 22 months of constant development. Debian GNU/Linux is a free operating system which supports a total of twelve processor architectures and includes the KDE, GNOME, Xfce, and LXDE desktop environments. It also features compatibility with the FHS v2.3 and software developed for version 3.2 of the LSB.

Debian GNU/Linux runs on computers ranging from palmtops and handheld systems to supercomputers, and on nearly everything in between. A total of twelve architectures are supported: Sun SPARC (sparc), HP Alpha (alpha), Motorola/IBM PowerPC (powerpc), Intel IA-32 (i386), IA-64 (ia64), HP PA-RISC (hppa), MIPS (mips, mipsel), ARM (arm, armel), IBM S/390 (s390), and AMD64 and Intel EM64T (amd64).

Debian GNU/Linux 5.0 Lenny adds support for Marvell’s Orion platform which is used in many storage devices. Supported storage devices include the QNAP Turbo Station series, HP Media Vault mv2120, and Buffalo Kurobox Pro. Additionally, Lenny now supports several Netbooks, in particular the Eee PC by Asus. Lenny also contains the build tools for Emdebian which allow Debian source packages to be cross-built and shrunk to suit embedded ARM systems.

Debian GNU/Linux 5.0 Lenny includes the new ARM EABI port, armel. This new port provides a more efficient use of both modern and future ARM processors. As a result, the old ARM port (arm) has now been deprecated.

Well i’ve been using testing-lenny for such a long time that this release doesn’t bring really something new or real joy stuff…but hey it’s a release. 😆

The Debian time frame of years between stable releases gets kinda disturbing at some point but you can’t really blame the Debian project because they have other goals in mind like stability, portability, longer support for a stable release and wide variety of packages already available and supported. If you want something different then you are always welcome to use other distributions like Fedora or Ubuntu which have a faster pace at releasing stable versions.

Tagged with: