Linux support on-site, on-line and in the cloud.

Firewalls, priorities and protection.

Firewalls, priorities and protection.

When I first started installing "firewalls" in the 1980’s they were very much seen in the context of forest fires and "fire breaks", a gap across which the "fire" could not travel. A hardened host was built which was connected to the internet and provided the services which you wanted to make available and only those services. IP forwarding was turned off. Access from the LAN to firewall was provided for maintenance and sometime for users to access proxy servers which could send out going traffic to the internet.

As firewalls became ever more complex software applicationsthere was quite a debate as to whether a firewall was necessary at all. If the only ports open were the ones on which you provided services, the crontrarian argument went, what was the point of packet filtering?

Well rightly or wrongly firewalls got more and more sophisticated and became the essential tool deemed necessary to protect the network. Other techniques, host hardening, perimeter networks became less of a priority. Without the firewall many organisation are now hopelessly exposed. The exposure of the enterprise to risk further exposes its clients, customers and partners similarly to severe risk.

Well configured and properly maintained firewalls are of course highly effective in keeping the undesirables out. The best firewall in world is useless however if its turned off!

I cannot even recall the number of times I have been told that a network communications problems "must" be down to the firewall. I was reminded of this recently when a client was experiencing problems with email and Courier POP3. It was a system that had been in place for some years. The system was built on a server and I had implemented an iptables
firewall a couple of years previously. The firewall prevented access to ports 110 and 995 except from the clients site and the hosts of the technical support staff. E-mail is crucial to the clients business and so a 1 and 1 consultant was brought in to accelerate the time to "fix". The consultant was quite clear and quite emphatic, the firewall was causing the problem.

It didn’t matter that the problem was intermittent, that the configuration of the firewall hadn’t changed, that the logs did not show any blocked POP3 traffic. It wasn’t relevant that netstat showed successful connection or that a sniffer on the network showed the relevant communications taking place. No the problem was the firewall. It was blocking traffic and needed to be turned off.

It reminded me of a similar situation when I was working for the putative "biggest bank in the world" in the City. I was managing 6 Checkpoint "Firewall 1" firewalls for the arm of the bank that dealt with exotic derivatives. The firewalls protected gateways to internet, extra-net gateways to Frankfurt, Hong Kong, San Francisco, New York, Singapore and Tokyo, and data feeds from Bloomberg and Reuters. When one of the data feeds went down it was only minutes before the panic buttons started being pushed. The trading floor was in uproar, the bank could be loosing hundreds of thousands of dollars within minutes.

I was asked to check the firewall. I did. Nothing from the data feeds was being blocked, nothing had changed on the firewall that day. We could see from the firewall logs that it wasn’t a problem, sniffing on the network interface could show that no traffic was arriving from the data feed source. Non of this held any sway with the network manager. It had to be the firewall and if it wasn’t well the only way to prove it was to take it down!

Well the long and short of it was that with tears of frustration in my eyes I ultimately carried out the instruction to "turn off" the firewall, it was that or clear my desk. And that was my priority you see, keeping my job.

The exposure of this multimillion pound enterprise and via it’s City of London hub the world wide corporation was less important than the priority of making profits on the trading floor.

Part of the problem I think is the demise of the systems administrator. The old systems administrator was responsible for everything, the servers, the network, telecoms, applications and system security. Because the systems admin was responsible for everything he had to understand it all and how each element fitted into the bigger picture. Now with so many functional divisions within a company’s IT provision, often, I am convinced, no one in the organisation really has a handle on how it all works together. The "networks" team primary ethos is to enable high speed communications. Firewalls get in the way of that process, firewall stop network traffic! This could be why they are so often identified as the prime suspect when things stop working, they are something of an anathema to "networks". The other side of the coin of course is that the network security team, working independently of "networks" and "production services" et al, tinker away at the firewall "improving security" often completely oblivious of the potential impact on the users or the business. They’ve only got to make a wrong call once or twice in year, to become the bête noire for the next ten.

Oh and in case your wondering, turning the firewall off made no difference to the data feed at all, the problem, as I knew, lay elsewhere. I have to say the network manager was big enough to approach me later in the day and explain what the problem really turned out to be, it wasn’t an apology exactly but it was much more than might have been expected. Were the banks systems penetrated while the firewall was down? No body knows, and I suspect nobody involved, wanted to know.