In adelphia.security-issues - article
<***@4ax.com>, on Tue, 08 Mar 2005
10:48:31 -0700, W. Mark Herrick, Jr. says...
Post by W. Mark Herrick, Jr.On Tue, 8 Mar 2005 02:17:11 -0500, Murray Watson
Post by Murray WatsonPost by W. Mark Herrick, Jr.As always, if you are seeing a good deal of ARP traffic from a
particular host, you're free to fire off a traffic log to my staff at
-MH
Has anyone requested UDP ports 1026-1029 be kept open? If not, you
could cut down on a lot of the ARPs we see by blocking them at the
borders. Clears up a lot of blinking lights and some network
bandwidth.
Noone has specifically requested those be open, but since the posts
are above 1024, there's a higher liklihood that they'd be used for
random UDP traffic flows, yes?
You have a point but while I'm probably not representative of most
consumers, I don't use any of the consumer oriented IM clients and
stuff like that, the only thing I show registering outbound on UDP in
1024-1029 are queries by some of my NTP clients. Out of 700 entries
over the last 6 months, 15 weren't originated from port 1024.
Any applications using UDP aren't too concerned about transport, the
kicker is a potential for DNS lookups to falter for a moment as they
skip over those ports.
So, it's a trade-off between a bazillion ARPs and an occasional
delay'd DNS lookup. But... one could either exclude Adelphia's DNS
servers from the rule or apply only at the borders.
In case this is not representative of what you're seeing in the glass
tower, here's a view from here. I have placed 5000 records of a
tcpdump on my web page.
http://users.adelphia.net/~mcwatson/watch.txt .
It is unretouched and contains a couple of internal packets.
You'll note that 5000 records were obtained in something just under
1.5 minutes, somewhere around 55/second. It's been like this since
the weekend, where before I rarely if ever saw any modem traffic that
wasn't initiated here. An occasional virus probe but not the steady
flow. It's possible that the messenger spammers updated their
netblock lists coinciding with the outage/changes noted here in
Mooresville NC.
That's 55 (packets/second)
x60 (3300/minute)
x60 (198,000/hour)
x24 (4,752,000/day)
x7 (33,264,000/week)
x4.3(143,035,200/month)
x365(1,729,728,000/year)
At ~1500 bytes/packet, that 660kbs of broadcast download bandwidth
consumed by ARPs
You'll note the somewhat sequential nature and double entry of the
ARPs, indicating transmission to 2 ports sequentially for each IP
(which corresponds with the 1026/1027 messenger spam sequences I'm
seeing in my logs) at sequential IPs at these times :
17:52:33.311831
17:52:39.071831
these are definitely Windows Messenger spammers and the list goes on
and on.
If your CMTS/routers with IDS/IPS could identify and partition the
sources, you'd help us recoup some of the bandwidth and traffic on
your backbone.
I think a good starting point would be to block UDP/1026-1029 from
APNIC netblocks at the border routers. Give that a try and see what
you end up blocking. China's hosts has been heavily compromised by
the Sql worms and are probably the biggest source of the messenger
spams, quite possibly not even forged IPs but one would have to do a
MAC trace to determine that, a tool I don't have in my arsenal.
You can see from the tcpdump would appear to contain several
neblocks, spanning more than my 69.164.0.0/18, more than
69.164.0.0/16
ie. 69.164.0.0 - 69.174.255.0.
not quite the 69.160.0.0/12 netblock and the 69.174.0.0 seems to be
small in number whereas 69.164.0.0/18 seems to be appropriately
represented.
Maybe the scope of the IPs is attributable to a typo in a
configuration. If Adelphia made some CMTS changes over the weekend,
I figured there would be some kinks to work out and it may take a
little while. Is the scope of these ARPs typical?
This rings as odd:
17:52:33.981831 arp who-has 67.21.36.246 tell 67.21.36.245
17:52:33.981831 arp who-has 67.21.36.246 tell 67.21.36.245
It would appear to be a router in Atlanta with an identity crisis.
Post by W. Mark Herrick, Jr.Post by Murray WatsonCurious, do the headends have the capability to egress filter not-
endemic IPs? If so, implementing that could offer some protection
within Adelphia's network from internally originated UDP/IP spoofs.
I'm also curious if the vendors are beginning to offer any stateful
IDS/IPS oriented features.
Headends - no. CMTS's (which may or may not be in the headend) - yes.
I can assume that's what you meant, though. :)
Yes. My use of headend is a generic MSO-end equipment word. I'll
start using the correct terminology.
Back to the meat of the question then, they are capable of
identifying an offending source based on rules or input from a
honeypot type device and either blocking or rate-limiting?
Post by W. Mark Herrick, Jr.Is it implemented? No - since we have no correlation between our
billing systems and CMTS's (directly), which would allow us to send a
signal to the CMTS, saying "this MAC is provisioned so it's good -
this traffic came from an unregistered MAC so drop it". Obviously,
this opens up some theft of service issues, but we've got other ways
of dealing with that.
Not at a MAC level, IP level. Certainly the CMTSs are IP smart, ie.
CMTS 123 manages 24.48.32.0/21 (and/or other netblocks). Giving it
the ability to reject any traffic initiated on the modem side with a
source IP != 24.48.32.0/21?? If it's not at the CMTS level then
surely at the next router level up.
While it's less significant and I can't say that I've seen it on
Adelphia but I recall an incident on RR where I was seeing traffic
from 1.2.3.4, an obviously invalid IP, one that ARIN had not and will
not delegated. Those packets should never have been seen, blocked at
the border routers, CMTS or other MSO equipment.
Post by W. Mark Herrick, Jr.We are currently working with some vendors to impelement some rather
interesting tools at the CMTS'ish level, that will (hopefully) be able
to ID and mitigate these types of malicious traffic, signal back to
the source (if it's internal), and then place the infected system into
a quarantine area.
Real cool stuff. :)
Good to hear they are proactively working at cleaning up what has
become a big mess over the last 10 years. With what I expect to be a
fair amount of processing power available on the modem, I'd think
they would be looking a pushing some of the IDS/IPS functionality
down to the modem level, via something like a configuration file.
Thanks for your attention and feedback, it certainly facilitates
learning for all participants.