TL;DR:
Donate via PayPal here (ignore the “AGDQ 2015” reference) or via Patreon.com/dwangoAC for a couple extra benefits. Thanks!

Hi, I’m dwangoAC and if you’re here it’s probably because you clicked on a donate link (thanks!). TASBot was born on December 22nd, 2013 when I first assembled him. It’s been a wild ride of ups and downs over the last decade but TAS event content I’ve led has raised more than $1.4m USD at various charity events and I aim to keep going. I supported all charity event travel costs with my day job, ultimately spending $20k more than I recouped through Patreon and elsewhere, but it was absolutely worth it. Unfortunately, I was laid off many months ago in a challenging time for the security consulting market and I’ve decided to focus on making new TASBot content instead. My kids (DefeneSam and Wrentendo) and I can’t relocate from the SF bay area and I need to net $6k per month just to survive due to high housing costs; I regret I’ll need to pause donating 10% on to charity for now but I’ll give you plenty of fun content if you can help us.

You can make a onetime donation via PayPal here (ignore the “AGDQ 2015” reference) if you’d like – thanks! (I’ve largely stepped out of the world of cryptocurrency but feel free to reach out if you’d like to donate in that way.)

Another way to donate that helps you get some nice perks is Patreon.com/dwangoAC which additionally gets you access to new content early with no ads and a shoutout in video content. It’s much appreciated!

I’ve been using ClusterSSH (cssh at the command line) to simultaneously interact with multiple remote systems for years. If you’re not familiar with why you might want to do this, have a quick glance at Kyle Rankin’s Hack and / article on ClusterSSH. ClusterSSH has been a very useful tool for me and it has some handy features such as being able to select and toggle groups of systems to send input to but I’ve always found the UI clunky and I wanted something that didn’t rely on using a GUI.

Enter tmux-cssh – it’s a wrapper that accomplishes the same thing but using tmux panes. The easiest way to get it is to git clone https://github.com/dennishafemann/tmux-cssh.git and run ./tmux-cssh (or copy the file to a location in your path). Here is an execution using the same username on each host, as hijacked from the documentation:

tmux-cssh -u my-user-name my_server 1.2.3.4 11.22.33.44 my_second_server my_third_server my_and_so_on_server

In my case, I’m using a custom port and have a couple servers that require a different username, so I’m launching it with the -sc option with quoted ssh arguments, such as:

tmux-cssh -sc "-p 1234 my-server-name" "-p 4321 differntuser@differenthost.com"

With either method, a tmux session will start in tiled mode with the number of remote servers you’ve connected to. At this point, anything you type will go to every server simultaneously. Once you’ve logged in to all of the systems you can now do fancy things like vim ~/.ssh/authorized_keys (which would select the home directory on all remote servers even if they have different usernames) and paste input to all of the remote servers at the same time. If one of the remote servers is out of sync or you need to send text to only one of them, you can hit your tmux escape sequence such as C-b (Ctrl+b) followed by : and type:

set-window-option synchronize-panes

This will change tmux to its normal behavior of typing in only one window at a time. It gets cumbersome typing the command out, so add the following to your .tmux.conf file:

# Toggle tmux-cssh all with C-b e
bind e set-window-option synchronize-panes

You can substitute a different key instead of lowercase e, but this one is free by default and I remember it with whether I want to send input to (e)very window or not. If you need to do something a bit more fancy you can also use tmux’s built-in window and pane movement commands to move panes to another window and operate on a group of systems at the same time. The synchronize-panes setting is unique to each window you’re looking at, meaning whatever you type only affects the panes / systems you can currently see.

This is already extremely powerful, but I’ve come up with another unusual use case – running the same commands on multiple local and remote sessions. The use case is when you need multiple instances of things that aren’t all remote systems – in my case, I’m using ipmitool to control multiple remote systems. My execution looks like this:

./tmux-cssh -sc "-p 1234 remotehost1" "-p 4321 remotehost2"

This will launch two remote SSH sessions. Create a new split pane with C-b % or C-b ” (these will be additional local panes on the system you’re running on). Once you have enough panes for your needs, hit C-b and the spacebar a few times to re-tile the windows. At this point, you can do something like this:

ipmitool -I lanplus -U root -P root -H 10.0.10.10 shell

Before you hit enter, back up to the last octet of the IP address and toggle synchronize panes, then type in the final digit of the individual remote hosts you want to connect to (say, 10.0.10.101, or rewrite the entire host as needed). This example is a bit convoluted, but can be of use if you have the same tool you need to run from two potentially separate networks. If all of your hosts are local you can just run tmux as normal and split panes then toggle synchronization. Once you are at an ipmitool prompt you can do things like send power reset to reset all of the IPMI-connected servers, or type sol activate to open up a Serial-Over-LAN interface on all of them.

There are a lot of other potential uses I can think of for some of these concepts and methods and I’m sure I’ll come up with more uses over time.

If you have an ISP like Sonic.net with an IMAP mail server that allows you to edit your .procmail folder, there’s a nifty trick you can use to move E-Mail into folders, i.e. without your client even needing to connect.  This can be handy if you subscribe to mailing lists that you’re interested in keeping up with but perhaps don’t want to be notified about at 2 AM.

Here are a couple of .procmail example sections:

:0: H
* ^Subject:.*(CRIU)
${DEFAULT}.CRIU/

This matches on all mail that comes in with a subject containing the string “CRIU” and places it in a folder named CRIU (which you should create before applying this rule). Saving the file is all that is required – the next incoming message will be silently placed in the folder without hitting your main inbox, and if you only monitor E-Mails in your primary inbox this also means you won’t get notified of new messages. You can use standard Procmail filtering methods – for instance, if you have a mailing list that always sends mail to the same address, you can do this:

:0: H
* ^To:.*(fspa)
${DEFAULT}.FSPA/

This will match all mail with the string “fspa” in the To: mail address. I’ve found this method of mail management to be extremely helpful and if you have a provider that supports this I hope you are able to take advantage of it as well.

I’m very intrigued by the idea of having my own personal server to play around with.  There was a good NBLUG talk about them in 2011 which piqued my interest and I’ve been messing around with one of my own for a while now.  I highly recommend them as a cheap way to get your hands dirty on a system you control and can meld to your will.  They’re a great alternative to jumping into something like an Amazon AWS instance and often come with cheap month-to-month pricing.  The cheapest servers are OpenVZ based which have fixed memory sizes and limited swap but are often all that is needed.  More expensive KVM instances are available but aren’t worth it unless you need the extra horsepower.

Note, however, that not all providers are created equal – some providers have a bad reputation for spotty network connectivity, excessive downtime, and other issues, so I’ll avoid recommending any particular provider for the time being.