Event management solution scaling - Practical example

As described in the previous blog post, every software; every server or every appliance has its limits.
Scaling beyond these limits is a task for an engineer to build something that can cope with the loads.
In theory one could adjust the open-source solution and live happily ever after, but in the real world.. well one has to deal with proprietary software or appliances and it's not easy to just migrate or replace it.

For such scenario, I've developed a small program called NFF that forwards the incoming traffic to several configured destinations. Currently it is built to listen on one port and forward it to several destinations, but with different configuration file it can run for several services (e.g. syslog; snmp-traps; netflow)

Note: in current version it only forwards the flows, but later on when protocol decoding is implemented, it would also be able to forward flows to specific destinations based on rules.

Integration would be done by installing this program on the same IP address that all systems send their logs/netflow/data to, and the appliance or software analyzing these would move to a new IP address.

In case the management decides to buy a bigger box or choose different supplier, this can be added to the distribution list during trial period in order to see if it fulfills the needs and expectations.

As I don't have a job where I could test this idea at scale, I hope some of you would provide me some feedback how well it can perform. I already have several ideas how to make it work faster..

Cryptography education

After a long break, I finally got back to writing yet another post (coincidentally, about what I did the last few months).
As cryptography is quite essential part of security engineering I decided to challenge myself by doing online course by Dan Boneh from Stanford called Cryptography I.
Originally I didn't expect that university courses go into much detail of how crypto algorithms are implemented in practice, but this course surely changed my opinion on practicality of university lectures. Of course I should have known that lectures from famous universities like Stanford are  expected to be worth the tuition fees.

Course content

Despite the practical nature of this course, it requires very good understanding of various areas of Math as well as the scientific method of lectures.
This actually was important part of practicality, as some areas of security engineering require or expect a mathematical proof of algorithm or process security. I'm not sure how such proof  would hold in front of an average auditor, but this course surely does good work proving weaknesses of explained cryptography functions.
The first course contains brief introduction with a short refresher in discrete math and goes right into stream and block ciphers (like DES or AES). Later it covers message integrity and hash functions and finishes the course with public key encryption (RSA and ElGamal).
The second course (expected to take place in June 2015) would surely bring more advanced and recent topics like elliptic curve cryptography or maybe digital currency concepts, so there's more to learn.
In most of the topic practical examples were given on how some services or protocols used the cryptography in wrong way.

Programming part

Besides the theoretical part it also included coding assignment, where the tasks were from encrypting content in efficient and secure way to deriving keys from insecure implementations. This experience would surely be very useful in code review or penetration testing work.
All the programming assignments were done in Python, with help of various libraries that offered the necessary crypto algorithms already implemented.

Conclusion

Everybody in the field of security who wants to call himself a security engineer should do this course (and pass), as the knowledge gained is very important in many areas of the security field. Whether one does security policy (to define acceptable crypto algorithms/key sizes etc.); ensures application security (performing code reviews or testing); provides authentication, file or storage encryption or builds VPNs, this course will help to understand what are the implications of choosing one or the other security algorithm and what would be the performance and security impact of these decisions.
With recent openssl vulnerabilities I can't stress enough how this knowledge improves the decision-making whether or how to deal with these vulnerabilities.

Heartbleed (CVE-2014-0160) vulnerability overview

Vulnerability description

OpenSSL released a bug advisory (CVE 2014-0160) about a 64kb memory leak in their library specifically in packet processing code for  the heartbeat extension (RFC6520).

The heartbeat consists of a request packet including a payload; the other side sends a response containing the same payload (plus some other padding).
In the packet sent by the attacker there is a payload size indicator, which is then used for sending the response packet back. Due to the fact that packet that was received is much smaller than specified in the payload size, the packet response function reads and sends the memory after the packet data.
This allows attacker to read up to 64kb memory without a trace (heartbeat is not logged) and can be repeated many times to increase the probability of containing valuable information.

More details (in FAQ format) can be found on a website created for this bug.

Vulnerability status of different versions of OpenSSL library:

• OpenSSL 1.0.1 up to 1.0.1f (inclusive) are vulnerable
• OpenSSL 1.0.1g is NOT vulnerable
• OpenSSL 1.0.0 branch is NOT vulnerable
• OpenSSL 0.9.8 branch is NOT vulnerable

Timeline

Bug was introduced to OpenSSL in December 2011 and has been in the code since OpenSSL release 1.0.1 on 14th of March 2012. OpenSSL 1.0.1g released on 7th of April 2014 has the bug fixed.

Testing

Some recommendations suggest evaluating the version of the openssl library via the client openssl version, but that only shows the library version and not necessarily version used by a webserver (can be statically linked).

Although there are many websites that claim to test this vulnerability (by using HTTPS protocol to access the given IP address), i would not recommend using them as storing the requests in the log would also create a nice list of vulnerable websites.

There is a number of offline tools and scripts that can do the same test as well:

• Tool coded in Go
• Tool in Python
• Tool in Perl
• Nmap script
• Metasploit module
• OpenVAS nasl wrapper
• Chrome extension
• Test using openssl client binary

There was also a mass-test performed on top 10000 sites, to see if they are vulnerable. Users of these websites should consider changing passwords/keys in order to protect their digital identity.

Vendor status

As the library is used in many products, I would focus on network vendors here:

• Cisco Advisory (for all products)
• Juniper Security Bulletin
• Palo Alto blog
• HP TippingPoint announcement
• F5 security advisory
• AWS security bulletin
• Fortigate advisory
• Checkpoint response

Note: The bug is in the code of heartbeat function, so I would expect it to be used in protocols or areas, where keep-alive is done. I haven't seen any information about other services like IMAP/FTP or others that use STARTLS.

Detection configuration

People at Sourcefire vulnerability research published the following IDS signatures for snort:

alert tcp $EXTERNAL_NET any -> $HOME_NET 443 (msg:"SERVER-OTHER OpenSSL SSLv3 heartbeat read overrun attempt"; flow:to_server,established; content:"|18 03 00|"; depth:3; dsize:>40; detection_filter:track by_src, count 3, seconds 1; metadata:policy balanced-ips drop, policy security-ips drop, service ssl; reference:cve,2014-0160; classtype:attempted-recon; sid:30510; rev:2;)

alert tcp $EXTERNAL_NET any -> $HOME_NET 443 (msg:"SERVER-OTHER OpenSSL TLSv1 heartbeat read overrun attempt"; flow:to_server,established; content:"|18 03 01|"; depth:3; dsize:>40; detection_filter:track by_src, count 3, seconds 1; metadata:policy balanced-ips drop, policy security-ips drop, service ssl; reference:cve,2014-0160; classtype:attempted-recon; sid:30511; rev:2;)

alert tcp $EXTERNAL_NET any -> $HOME_NET 443 (msg:"SERVER-OTHER OpenSSL TLSv1.1 heartbeat read overrun attempt"; flow:to_server,established; content:"|18 03 02|"; depth:3; dsize:>40; detection_filter:track by_src, count 3, seconds 1; metadata:policy balanced-ips drop, policy security-ips drop, service ssl; reference:cve,2014-0160; classtype:attempted-recon; sid:30512; rev:2;)

alert tcp $EXTERNAL_NET any -> $HOME_NET 443 (msg:"SERVER-OTHER OpenSSL TLSv1.2 heartbeat read overrun attempt"; flow:to_server,established; content:"|18 03 03|"; depth:3; dsize:>40; detection_filter:track by_src, count 3, seconds 1; metadata:policy balanced-ips drop, policy security-ips drop, service ssl; reference:cve,2014-0160; classtype:attempted-recon; sid:30513; rev:2;)

alert tcp $HOME_NET 443 -> $EXTERNAL_NET any (msg:"SERVER-OTHER SSLv3 large heartbeat response - possible ssl heartbleed attempt"; flow:to_client,established; content:"|18 03 00|"; depth:3; byte_test:2,>,128,0,relative; detection_filter:track by_dst, count 5, seconds 60; metadata:policy balanced-ips drop, policy security-ips drop, service ssl; reference:cve,2014-0160; classtype:attempted-recon; sid:30514; rev:3;)

alert tcp $HOME_NET 443 -> $EXTERNAL_NET any (msg:"SERVER-OTHER TLSv1 large heartbeat response - possible ssl heartbleed attempt"; flow:to_client,established; content:"|18 03 01|"; depth:3; byte_test:2,>,128,0,relative; detection_filter:track by_dst, count 5, seconds 60; metadata:policy balanced-ips drop, policy security-ips drop, service ssl; reference:cve,2014-0160; classtype:attempted-recon; sid:30515; rev:3;)

alert tcp $HOME_NET 443 -> $EXTERNAL_NET any (msg:"SERVER-OTHER TLSv1.1 large heartbeat response - possible ssl heartbleed attempt"; flow:to_client,established; content:"|18 03 02|"; depth:3; byte_test:2,>,128,0,relative; detection_filter:track by_dst, count 5, seconds 60; metadata:policy balanced-ips drop, policy security-ips drop, service ssl; reference:cve,2014-0160; classtype:attempted-recon; sid:30516; rev:3;)

alert tcp $HOME_NET 443 -> $EXTERNAL_NET any (msg:"SERVER-OTHER TLSv1.2 large heartbeat response - possible ssl heartbleed attempt"; flow:to_client,established; content:"|18 03 03|"; depth:3; byte_test:2,>,128,0,relative; detection_filter:track by_dst, count 5, seconds 60; metadata:policy balanced-ips drop, policy security-ips drop, service ssl; reference:cve,2014-0160; classtype:attempted-recon; sid:30517; rev:3;)

Note: these signatures report only heartbeat use with large data size, not actual ex-filtration of  passwords or other sensitive content. So it is useful to detect vulnerable servers in order to patch them. Also they don't inspect any other services than https that might be vulnerable.

Mitigation

There are several options that can mitigate this vulnerability:

• upgrade openssl library to 1.0.1g or higher (or downgrade to 0.9.8)
• compile openssl library with -DOPENSSL_NO_HEARTBEATS option
• use different ssl library 
• use perfect forward secrecy (this does not prevent leaking of the session keys or other memory content)
• drop packets with the heartbeat requests (or heartbeat packets larger than normal size)

As this vulnerability has potential to disclose keys and passwords used, it is recommended to change the passwords and/or generate new keys used on external websites or services using SSL in the time-frame when this code was used (since Dec 2011).