A few days ago, we got a backup Internet service at home. I connected both the DSL and the cable modem to the same laptop, configured the routes with weights according to the bandwidth expected from each of them, configured Tor exit node to use only the address where it has not given me headaches, adjusted some scripts to detect port shuffling at reboots, and watched as it worked just as I expected.

Well, almost. The new service also assigned an IPv6 address to the interface connected to it. The old service was supposed to support IPv6, but somehow it would never be assigned. Turns out that, with the new service in place, the old one also started assigning IPv6 to its interface.

I didn't used to monitor the old interface very much. I used to have only SSH and Tor ports open, and the new service did not even offer incoming connections (thanks OpenVPN and Tor's Onion Services for independently enabling me to get in from the outside), but did IPv6 change it?

Though tcpdump is my friend, two ports and all of the noise from Tor and from regular family use makes it all a little more interesting to monitor. Nothing jumped at me for several days.

The other day, my primary laptop seems to have overheated and died. Next day, after I failed to bring it back with parts from another dead sibling, I got its disk and RAM onto another old backup laptop to go online, and then another similar old laptop that plays the role of home gateway spontaneously restarted. Maybe a power fluke and a bad battery, but... Weird...

I hadn't checked that both services had been brought up properly after the reboot, but something was working. I woke up today with "Internet down". Some cables had mysteriously moved between the modems, but that was one of the cohabitants's attempts to troubleshoot. I put the cables back in place, checked that the automated configurations picked them both up, and... What's this weird connection here? Tor was still down, and everyone else had connected directly to one of the modems over WiFi!

I kept Tor down, but failed to find any records of the weird connection. Tcpdump could see it, but lsof, netstat and ss all failed to report it. And it was not just one, apparently there were several of them. Well, that explained the spontaneous reboot from the other day. Some scriptkiddie was in, apparently using it as part of command and control infrastructure for a botnet. What a lucky week! (not!)

I tried to find evidence of any changes to the filesystem without disturbing the running system, taking a snapshot of root and hoping whatever the scriptkiddie used to hide their presence would not pay attention to snapshots. Nothing. I tried to compare it with an earlier snapshot, but that was from before a recent upgrade that was long overdue.

I figured I'd reboot into one of the pre-upgrade hoped-good snapshots and look for signs of an intrusion on the new system, and for any signs of it with the old system. That went well. The gateway rebooted, fixed some DNS issues because of local slave maps that had expired and wouldn't update, installed updates and rebooted, all while monitoring for the telltale packets that had confirmed the intrusion. None whatsoever. So far so good.

I'd also reset both modems, so that they'd pick up new dynamic IP addresses. With the gateway rebooted into the new kernel and updated system, with all network ports enabled, I moved back in front of my primary laptop to keep on monitoring and investigating from a more comfortable place. GNU jami had given up trying to reconnect and quit; the Tor proxy on it took a little while to give up on the old circuits and set up new ones, but everything seemed to be back to normal.

Not so fast! Once I started connection monitoring on the gateway again, I noticed a few unexpected connections there. One by one, I looked for them in lsof or netstat, and found them. Phew. Until I reached one that wasn't there. An ongoing TCP connection, sending smallish packets every few seconds and receiving shorter responses. Triple-checking lsof, netstat and ss, I confirmed it wasn't there. Oh my...

Ok, let's try to figure out where this is coming from. I hoped that, by breaking the connection, I'd get whatever was behind it to reconnect, and then the audit trap I'd put in place would catch the culprit. Weirdly, my attempts to reject the outgoing packets were unsuccessful, so I ended up routing them to a blackhole and waiting for the connection to timeout. Eventually, it did, but nothing took its place.

But wait, now there's a suspicion connection to the Linux-libre server! Oh, my; it will suck if that's compromised too. Cleaning up both my home network and that remote server will be quite a pain. I wonder if the kiddies used that server to break into my place, or vice-versa... It's embarrassing either way, but... Gotta deal with it.

One of the two connections was listed in lsof and netstat on the gateway, it was an openvpn tunnel. As for the other... Let's see if the connection is hidden on the Linux-libre server too. Hmm, that port number is familiar, it's where a chrooted ssh in a buildroot listens on. Hey, look, my laptop has automatically reconnected to it!

Very long story short, the reason some packets didn't show up in the gateway's lsof or netstat output was that they came from other machines, and in all that stress, I didn't even think of checking /proc/net/nf_conntrack for masqueraded connections. The other suspicious TCP connection was forwarded by the Tor proxy running on the laptop! And once I restarted GNU jami on the laptop, I started seeing the other suspicious UDP packets fly by, with the various ASCII keywords with which I recognized them before. Its source port did not appear in netstat or lsof on the gateway, nor on the laptop, but it did appear in nf_conntrack, and there I could find the original source port, prior to masquerading, and confirm that the laptop had that port in use by jami. Phew!

I had found the scriptkiddies, and they are me!

Enough of following ghosts.

PS note to self: do not conclude there's a breakin with patched syscalls just because a gateway won't show in lsof or netstat a port that it is merely forwarding.

PS note from scriptkiddie: do not conclude there is no breakin just because you couldn't find it yet.

So blong...