Vote for Katz! When the election comes, you may as well vote for Katz!

How does Websense suck? Let me count the ways!

I've never been a big fan of web filtering technology. I like my porn and pirated software.

I first ran into issues with Websense back when I was working at a former employer who used it. I wanted to view @alecmuffett's blog and couldn't get through. Alec was kind enough to chronicle his conversations with Websense over how he was classified as "hacking" (evil) vs. "security research" (legit.) In fact, they misclassified him both in March and December of 2004.

So today I went to read @Av8rDan's latest blog entry. For some reason Websense has him classified as "sex."

I know that aviation-talk can be a valid substitute for sex for many of us pilot and pilot wannabes out there, but I don't think that is Websense's intention with this incorrect classification.

So I went back to websense.com to try and make sense of this and to see how we can get Dan's site fixed. He runs an advertising firm dedicated to serving the aviation community, and potentially this could hinder clients from reaching him and his blog. Incorrect categorization is common, and Websense even has a publicly-accessible knowledge-base article on the subject.

The problem is with Websense's process for submission of a recategorization. Back when Alec had to deal with it, a web-form led to an e-mail exchange (I think.) Now, 5 years later, you must register for a support account with Websense to search for the URL and then suggest that it is incorrectly categorized. That in and of itself doesn't seem to be a big deal. The problem comes from the fact that in order to register for Websense support you need a valid product key. That means you have to be a paying customer (or perform an additional registration for an evaluation kit) to sign up to flag a URL as being improperly categorized. Plus, once you submit to the support portal with a valid product key, having your support account added is a manual process that takes up to 24 hours. So you must wait an additional day before being able to submit a miscategorized URL.

Editors note: As we later found out via twitter there is an e-mail address to submit to for reclassification. However, at least when I looked at it, this e-mail address is not published anywhere. I wound up getting the info via twitter from someone who (I think) works for Marketing/PR at Websense. suggest{spam.trap}@websense.com .... of course, take out the {spam.trap} (I'm trying to be nice so spiders won't pick it up.)

Speeding up OSX Cocoa apps…

If a OSX app is Cocoa and it is not optimized for specific programming libraries you can speed it up. I did this for Adium and it is a lot faster now.

Here is the famous apple tech note thanks to the Firefox dev team.

The trick is to turn off some graphics throttling in a configuration file. Here is how to do it...

Navigate to the app via finder & show contents

Find the Info.plist file and make a backup...

Edit the info.plist file. You'll add
<key>CGDisableCoalescedUpdates</key>
<true/>
ahead of the line that says "CFBundlePackageType"

Save your work and restart Adium (or other app in question). Boom. Faster graphic updates.

Post-it all over!

One of the things beaten into us in CISSP training is that it is important to have a corporate security awareness program. If people don't know to be secure they won't be.

As we've learned from society over the past 200+ years having a good propaganda machine is a good way to get people to do what you want. Therefore over the past 10-15 years companies have developed security posters.

Two years ago over at Defense Tech they posted some posters as well as likes to WWII posters and some udderly-bad posters. I've also found three more sites with useful stuff. The UK Infowar site, DHS/CERT, and Central Coast Security all have some good PDFs for download.

What I'm really looking for is a "pick strong passwords" poster that fits a Soviet-era propaganda poster motif for our break-room. Any ideas?

Siebel Security Basics

Preface

My day job has given me some fantastic exposure to Siebel from an infrastructure and application management point of view. My personal opinion is that the application suffers by trying to be too configurable and too robust. In an effort to appeal to be a single solution for many different business scenarios the product becomes bloated and burdensome. If I was building a CRM solution for an enterprise I'd use a completely different set of tools.

In the deployment and management of the application I've learned a lot about how it works and how it can be broken both from a security and performance standpoint. Rather than say "here is a 0-day exploit" my goal is to highlight a few bugs, features, and fun facts, and show how easily information about the application and the way it is deployed can easily be found.

I can go into a diatribe about Siebel's architecture limitations, and I may do that in another blog entry in the future. In short, unless specific steps are taken some really ugly SQL can be auto-generated when coding in Siebel. Faults like lack of bind variables, no limits on results returned, etc. One end user doing a custom query can bring an entire enterprise to its knees.

Back to the task at hand; there are a few documents that exist about securing Siebel, but most of that documentation is horribly generic when it comes to the environment surrounding the application (useful tips like removing .rhosts files) or delves deeply into the management of the application (change the default passwords.) This document intends to be specific about the security surrounding the application.

Introduction: What Is Siebel?

Siebel is a CRM application framework. It provides an integrated way for internal and external customers alike to access and manipulate data. When you call into a call center (like as a DirecTV subscriber, for instance) the custom application that the CSRs use to manage the call, your satellite service and billing is handled through this application.

One reason people like Siebel is because it provides a robust way to interface with different, custom applications external to the Siebel infrastructure. In the case of DirecTV those custom applications push ACLs to the satellite transmitter to enable your receiver. (As a DTV customer I assume this is how it works.) Siebel has hooks for MQ Series as well as mainframe screen-scraping software.

Additionally, end-users can pay their bill through a Siebel-based application and the record of that payment is managed and viewable by an internal support employee. The data on the back-end is (usually) contained in a single database, but Siebel has different modules, known as Business Applications which can be used to format, present, and manipulate the data that is most appropriate. I only highlight DirecTV as an example because they're one of Siebel's best customers and there are a number of press releases proclaiming this.

There are at least 74 different business services which include canned solutions for energy, financial services, car dealers, and healthcare verticals. Although you may not see a lot of these Siebel Business Applications externally on the internet many companies use these internally.

With the development of the Department of Homeland Security my Siebel Administrator friends would joke about the proliferation of Business Applications and the creation of the “eterrorist” module, referencing the thoughts of Tom Siebel and a Lockheed Martin press release.

Below is a list of many Siebel Business Applications:

ecustomer	erm	ermadmin	ermemb
emarketing	etraining	eevents	prmportal
prmmanager	esales	eauctionswexml	eservice
sales	callcenter	cra	service
marketing	wpsales	wpserv	wpprm
wpeserv	eai	eai_anon	servicece
salesce	smc	edealer	edealerscw
econsumerpharma	eprofessionalpharma	esitesclinical	ecommunications
emedia	eenergy	eautomotive	ehospitality
epublicsector	pseservice	econsumersector	eretail
eaf	sismarketing	echannelaf	echannelcg
echannelcme	econsumer	eEnergyOilGasChemicals	eCommunicationsWireless
epharmace	medicalce	cgce	siasalesce
siaservicece	epharma	eclinical	emedical
ucm	fins	finsconsole	finsechannel
inseservice	finsesales	finsebanking	finsebrokerage
finseenrollment	finsecustomer	finssalespalm	esalescme
ecustomercme	htim	htimprm
loyalty	loyaltyscw	eloyalty

Introduction: Siebel Basics for Infrastructure Engineers

Siebel is laid out in such a way that it is conducive to fit a “textbook” internet application architecture. It allows for web services to be put in a DMZ, a middle-layer with dedicated Application servers and a backend that holds the Database. Although these paradigms make for pretty Visio diagrams that will make most security departments swoon some basics about how the application is deployed makes it vulnerable to a lot of information gathering and DoS attacks.

Siebel provides a lot of support for additional languages. If you're going to be a global company it helps to speak more languages than just English. Siebel takes advantage of the ISO 639-2 3-letter spec for languages. ENU is for English, ESN for Spanish, etc. Below is a list of a few languages Siebel supports:

EUQ Basque	CAT Catalan	CHS Chinese (Simplified)
CHT Chinese (Traditional)	SHL Croatian	CSY Czech
DAN Danish	NLD Dutch (Standard)	ENU English (American)
FIN Finnish	FRA French (Standard)	FRC French (Canadian)
DEU German (Standard)	ELL Greek	HEB Hebrew
HUN Hungarian	ITA Italian (Standard)	JPN Japanese
KOR Korean	NOR Norwegian (Bokmal)	PLK Polish
PTB Portuguese	PTG Portuguese	RUS Russian
SKY Slovak	SLV Slovenian	ESN Spanish (Modern)
SVE Swedish	THA Thai	TRK Turkish
PSE English (Pseudo)	PSL English (Pseudo

To create a useful paradigm Siebel appends the language to the Business Application after a "_". So the "esales" Application in English will be "esales_enu". This creates many potential targets for a possible cracker or security professional trying to evaluate an environment. Most sites only install the language packs they require.

A typical Siebel session is initiated when a user connects to a web site. The Siebel extensions, a set of C/C++ libraries that hook into the web server software (IIS, iPlanet, Apache) are invoked when a specific URL is reached. This Siebel Web Extensions open up a connection to the Application Server to figure out what to do. That application server will process what is sent to it, initially just a user name and password. Depending on the configuration this can plug into an ADSI model or LDAP or even an Oracle Database. Once authenticated a user has only the access to the application that is granted to them. The Siebel terminology for this is a "view."

This is a fairly simple model. Larger enterprises will place load balancers ahead of web servers and between the web and application servers for redundancy and availability. The Siebel Web Extensions also have the ability to round-robin between application servers. Application servers themselves can be configured to perform different duties. This is managed within the Siebel application and specifically by a single-point-of failure called a Siebel Gateway Server. Larger sites will cluster their gateway servers to protect against this SPOF.

Siebel measures its “work output” in “activities.” For every activity there is a corresponding entry in the database. An activity can include inbound and outbound calls, the escalation of a call to a manager, or a login request. This can include any self-service transactions through web portals, too. These activities can be linked together; a CSR answering a call can see that a user failed to properly execute an on-line transaction (pay their bill.)

Specialized application servers can include servers that handle specific processing of batch duties or a specific class of activities. There are also servers that plug into Microsoft Word for the dynamic generation of form letters, survey forms, or anything else that comes to mind.

The specific configuration information of “what activity occurs where” is managed by the Siebel Gateway Server and is kept persistent between application restarts through the “siebns.dat” file. If this file is lost the entire enterprise has to be reconfigured and likely re-installed.

Another paradigm of a Siebel environment is the “Siebel file system.” This is a Windows share (or NFS if run on Unix) that allows consistent static data to be shared among application servers. This data can include some of the content generated by the document server referenced above as well as the preferences for different users of the application (window sizes, number of columns displayed, favorite searches, etc.) The siebns.dat file is also stored in this location.

Security of this fileshare is obviously important. The Siebel Service (Windows) or Application Server daemon (Unix) needs read/write access to this share. It’s important that a specific daemon user be created on either platform that limits the rights to this share.

To keep things interesting Siebel has a user (the Anonymous or Guest user) that is used to generate the login page. This user exists within the authentication database and application and only has access to the view which generates the login page. This provides some security because a web server needs to store these credentials in order to create a login page and generate a web-based session. Without the credential information no-one can place a dummy server into the mix and perform any type of man-in-the-middle attacks. Additionally the content that can be accessed and displayed by the Anonymous or Guest user is limited and is managed by application managers.

Officially the Siebel Web Server does NOT store any static data for the long-term. It fetches the most current application data from the Application servers when the first connection to the Siebel Web Server is invoked. The Web Server then caches this data in the $SIEBEL_ROOT/public directory. The "public" directory receives most of its information from the corresponding Siebel "WEBMASTER" directory on the application servers. This cache can be refreshed with specific Siebel commands to the web server. However, this cache is never deleted. Once downloaded the files continue to live on the web server unless it is manually deleted.

Upon a server restart a "new" initial connection will cause these files to be over-written. If these files are "old" and newer files are in play, the old files will still continue to live on the Web server disk.

Remembering how Siebel depends on at least one language being installed that language will map to its own "public" directory. It's possible to have a $SIEBEL_ROOT/public/fra and $SIEBEL_ROOT/public/enu.

By default the Siebel Web Extensions are only invoked when "start.swe" or "*.swe" is called. There is no "start.swe" file on the disk. If you want to show-off you can connect to a Siebel application using "http://server/esales_enu/start.swe.Frankie_Say_Relax" and the standard Siebel Application will still paint. The most useful default Siebel account is called “sadmin.” This is the Siebel superuser account. This account can be changed and renamed.

Like most applications there are different text-based configuration files to keep things in check. For the purposes of this document the major configuration file to worry about is the “eapps.cfg” file. This is a straight text file that is used on the web servers. It maps the incoming requests to specific application servers or to a load balancing mechanism. It specifies defaults for certain timeouts. It is used to identify what ports Siebel needs to know about and it stores the login information and passwords for the Anonymous and Guest accounts. It does use a hashing function to store the passwords in the flat text file.

Exposure: Information Gathering

With the information presented above finding specific Siebel servers isn't too difficult. All someone has to do is google for "esales_enu" or any of the other possible combinations of Business Application and Language to find a URL. Alternatively sites can change from those defaults, but a "start.swe" may still be cached by google.com if it follows a re-direct. Therefore alternately googling for start.swe will find other Siebel installations.

Typically the Siebel configuration will create a redirect from "http://server/esales_enu/" to the "start.swe" URL. However, the "http://server/esales_enu" maps directly to the "public/enu" directory on the web server files are kept. There are scores of default files in the "public/LANGUAGE" hierarchy that can be used to determine what exactly is running.

To help administrators keep track of what version and patch revision of the application is deployed Siebel uses a few simple text files. All Siebel deployments will have a "base.txt" somewhere. Siebel actually puts this file in MANY places. One of the default places it is put is in the "public/LANG" directory of the running web server. All someone has to do to find out what specific version of Siebel is being run is "http://server/esales_enu/base.txt" and read the file.

7.5.3.15 [16279] LANG_INDEPENDENT patch applied.
HOTFIX QF0F21

Aside from "base.txt" there is also a "LANG.txt" which will look identical (or it should be, as the only supported configuration is to have the Language and SWE itself patched to the same revisions!)

Also in the "public/LANG" directory is an "About_Siebel.htm". This provides typical lawyer-license information and less specific version information.

Under "public/LANG" There are other typical directories such as "files" which holds items like CSS files and "images" which hold images. There is even a "help" which holds all kinds of defaults on how to use the application and possibly custom help files for that specific application deployment. There is no "default.htm" or "index.html" in the help directory but there is a "siebstarthelp.htm" and "siebindex.htm". There are other directories, too.

Extrapolating from our "base.txt" there is a number between square brackets. That's the build number of the specific application in-use by the Siebel Installation. That number also corresponds to a directory on the disk which holds specific custom, auto-generated javascript in-use by the application. This is the meat and potatoes of user interaction with the Siebel Application; it controls what fields are used and how they're autogenerated and shaded.

Exposure: Potential Risks

Combine the above knowledge of the location of the application’s javascript and a web server that has directory browsing enabled and there is a cornucopia of information that can be used and abused by someone who is interested in gathering information and formulating an XSS attack.

Note that these filenames are fairly consistent across all Siebel deployments regardless of the Business Application in use.

Another worrisome concern is the use of the “eai_anon_LANG” business application. This is primarily used for back-end processing of incoming and outgoing data streams used by Siebel and other outboard applications. If connections to the eai_anon_LANG business application are not encrypted anyone with a network sniffer can find out application authentication data and the integrity of the environment can be compromised. This is really true for any Siebel application that is using http and not https; any sniffer can find the HTTP POST traffic and snag a username and password.

Conversely, if someone outside the immediate enterprise infrastructure has access to the client of the “eai_anon” business application (say a business partner or client with a compromised system) they can watch the outgoing traffic even if it is encrypted over the wire.

With this information an attacker can gain ways to authenticate to the system as well as manipulate the data without interfacing with the Siebel user interface.

Exposure: Denial of Service

Before being acquired by Oracle, Siebel did 99% of its development on Windows systems and then used automated tools to move it over to Unix. The tools are kind of braindead and stick to basic ANSI standards.

Code was compiled to be 32 bit clean and not 64 bits. This means code is limited to 32 bit C defaults for basic constraints such as FD_SETSIZE (1024) and the number of file descriptors in use by stdio.h (255). The Siebel Web Extension code is limited to defaults that were created in BSD 4.2 and AT&T V6 that ran on a VAX 11/750. These limits are not overruled by the use "rlim_fd_max" or "rlim_fd_cur" on Solaris, either.

Unless a business is running a Siebel version of 7.8 and greater these limitations exist. For versions earlier than 7.8 unless a specific fix-patch is applied (a QFE patch, not a standard patch) the limit is there. For more details see Siebel Alert 1175.

Note that a Unix web server is required for the DoS because of how the code is ported over. Not having access to an IIS server or knowing the limits from MFC/MSFT C++ installation I have no way of testing for a DoS condition on Windows.

Different Unix flavors have different compile time limits imposed on the Siebel Web Extensions. From what I can gather FD_SETSIZE is 2048 for HP-UX and only 1024 for Solaris. There may be limits on Windows/IIS web servers, too, but I am unaware of what the defaults for MFC/C++ are.

All it takes is surpassing those stated limits for a Siebel installation to become unstable. Using the techniques outlined above for finding files on the Siebel Web Server 1025 concurrent requests for various files is all it should take to make a Siebel installation unstable and possibly crash the Siebel Web Extensions. The only way to recover from this type of crash is to restart the Siebel web server, severing the existing connections and interrupting the usage of the system.

Solutions and Practices

As an administrator of servers running Siebel I've learned the following that can prevent the above Denial of Service and Information Disclosures:
• A little security through obscurity can mask what software is being run. Changes to the default login page so it doesn't have the Siebel logo, re-writing eapps.cfg and obj.conf or httpd.conf to use custom directories will easily mask what application is running from various end users. Deleting the base.txt and LANG.txt files from the web servers and the application servers so they're not re-populated.
• Turning off or renaming the _stats.swe which is done through eapps.cfg
• Deleting the default help files if they're not required, both from the web and application servers.
• Turn off directory browsing on the web server.

That clears up your information leaking from a would-be attacker. Preventing the potential DoS through compile limits involves patching your Siebel installation. Researching via Siebel's "supportweb" Alert 1175.

For those who want to learn more Siebel/Oracle has a lot of documentation on-line with their Siebel Bookshelf.

Securing Siebel Web Servers

I haven't seen a single piece of documentation on locking down a web server that has Siebel extensions installed. I'm going to put a quick summary together here to help the internet at large out. This guide will mainly serve iPlanet/Sun One web servers on Solaris with Siebel 7.7, but a lot of the information is applicable elsewhere, too.

1) Install your web server as normal.

2) Tweak your web server prior to the Siebel install. That means install SSL on it first, tweak RqThrottle to 1200 in magnus.conf (Siebel recommended.) Also tweak your box so that file descriptors aren't an issue. On a Sun box that means add the following to /etc/system:

set rlim_fd_max=64000
set rlim_fd_cur=8192

3) After the tweaking is done then install the Siebel application which plugs into the web server (commonly know as the SWE, SWSE or Siebel Web Extensions.) Once that is done stuff gets to be more fun.

4) By default Siebel has weak permissions in the $SIEBEL_ROOT/ directory. Check to see that items aren't world-writable. Note that the $SIEBEL_ROOT/log directory should be read/writable by the user that your web server runs as. The same with $SIEBEL_ROOT/public.

5) These days Siebel automagically encrypts the user passwords in $SIEBEL_ROOT/bin/eapps.cfg. Make sure they're encrypted. You can also chmod 600 them and make sure your web daemon user is the owner.

6) Also related to eapps.cfg and eapps_sia.cfg edit out the Siebel business components (aka sub directories/web paths) that you don't need. It won't hurt anything.

7) The SWE install tweaks the web servers obj.conf file, too. Also take out the extra components in there; just leave the ones you need.

8. eapps.cfg has functionality that will allow for "rolling" logfiles. This is useful because Siebel Web Extensions are handicapped with 32 bit libraries at compile time. If you have a large web server that means the files in $SIEBEL_ROOT/log will be limited to 2GB. Use what settings work for your environment, but examine the following directives that can go into eapps.cfg

LogSegmentSize = 1000000
LogMaxSegments = 5

9) If someone is being creative and accessing your server via an IP address and you want to force their browser to re-write the URL to use your web site address this can also be accomplished through the eapps.cfg file. This is done through the "EnableFQDN" and "FQDN" parameters. It's pretty self-explanatory.

10) eapps.cfg can be used to force the web server's default page to invoke the Siebel Web Extensions. This means you don't need to manually redirect users from "http://your_site" to "http://your_site/eservice_enu". Just change the entry to [/] and then make sure your DocumentRoot is set to $SIEBEL_ROOT/public/enu (or whatever your language happens to be.)

That's really it.