The Basic "IF"

IF functions in MySQL (when used inline, as opposed to IF in a stored procedure) are nothing more than ternary statements. They take three parts and have a very simple syntax:

IF(condition, value if true, value if false)

So for example, if you wanted to calculate the sales tax on an order, but only if the purchaser was in Pennsylvania:

SELECT IF(state = 'PA', price * 0.06, price * 0.0) AS tax
FROM order

Nesting "IF"

The example that I recently encountered was needing to conditionally determine a person's location, in order to sort results by location.  The problem is that within this application, the business rules are such that a person can have:

• one or none of (home USA address or home international address)
• one or none of (work USA address or work international address)

... AND we need a query that selects all people ordered by location.  We don't have a single field to order by.  This application separates US and international addresses to better enforce data constraints and validation, so we end up with field names like home_us_state_abbr, or work_int_territory in various tables.

It's basically an inheritance pattern where all addresses share common fields, yet have additional fields dependent on the type of address (US or international). 

So, I can't select all people and hope to sort just by home state, because home state will be null for those people who either don't live in the US, or didn't provide a home address at all.  For this application, a person's generic location is assumed to be the first non-null result found in the following list:

• home USA address
• home international address
• work USA address
• work international address

...which is practically pseudo-code for our query. I'll spare further details – I imagine you get the point and will understand the field names used here, but the nested IF we ended up using in the query is:

SELECT
IF(home_usa_state_abbr IS NOT NULL, home_usa_state_abbr,
   IF(home_int_territory IS NOT NULL, home_int_territory,
      IF(work_usa_state_abbr IS NOT NULL, work_usa_state_abbr,
         IF(work_int_territory IS NOT NULL, work_int_territory, 'zzz')
         )
      )
   ) AS location_sort
...
ORDER BY location_sort

Essentially, the first non-null value encountered is assigned to the alias "location_sort". Of note is the use of 'zzz' on line 5, which essentially relegates those people with no specified address to the end of the result set. Without this people with no address were originally listed first, which was undesirable.; (See stackoverflow's SQL how to make null values come last when sorting ascending for a different way to accomplish this when you're not already selecting for a sortable value.)

Now, If using ENUM were our only option for representing what continent a country belonged to, we could all move on to debating the merits of NoSQL or whether Git was better than SVN, or why your favorite framework sucks. But there is a superior option to ENUM in most cases: the lookup/joined/reference table. (We'll call them reference tables.) It's pretty simple:

Wikipedia describes a reference table as:

...a table into which an enumerated set of possible values of a certain field data type is divested. For example, in a relational database model of a warehouse the entity 'Item' may have a field called 'status' with a predefined set of values such as 'sold', 'reserved', 'out of stock'. In a purely designed database these values would be divested into an extra entity or Reference Table called 'status' in order to achieve database normalization.

So, with reference tables available as an option to represent an enumeration, let's look at why the ENUM data-type is evil:

1. Data isn't being treated like data.

Male/Female, Mr/Mrs/Ms, Africa/Asia/etc: these bits of text that people use ENUM columns for are data. When you use an ENUM column, you're technically moving data from where it belongs (in actual database fields), to somewhere it doesn't (into the database metadata, specifically a column definition). This is different than putting constraints on the data, which is what we are doing when we say that a numeric column can only hold an integer, or that a date column can't be null - that's fine and quite necessary. With an ENUM we're actually storing pieces of data in a place that was only intended to hold crucial information about the model. In short, an ENUM column violates the rules of normalization. This may seem academic or pedantic, but it is actually why a lot of the other reasons on this list can be problems!

2. Changing the member list of ENUM columns is very expensive.

Invariably, what happens is this: you create an ENUM column and say "no way NEVER will this list change or need added to". But humans are really poor at estimating the entire scope of something, and even worse at predicting the future. R&D dreams up a whole new product type. Your company adds another shipping method. North America crashes into Asia.

The problem is that changing the member list for an ENUM column restructures the entire table with ALTER TABLE, which can be very expensive on resources and time. If you have ENUM('red', 'blue', 'black') but need to change it to ENUM('red', 'blue', 'white'), MySQL needs to rebuild your table and look through every record to check for the now-invalid value 'black'. MySQL is literally dumb and will even do this when all you did was add a new value to the end of the member list! (It is rumored that appending an ENUM member list will be handled better in the future, but I doubt that this is a high priority feature.)

A full-table rebuild may not cause much pain on a small table, but on a large one it is possible to peg your resources for a long time. If you use a reference table instead of ENUM, changing the list is as simple as INSERT, UPDATE, or DELETE, laughably-cheap operations by comparison. It's also important to note that when altering an ENUM member list, MySQL converts any existing record values that are not included in the new ENUM definition to ' ' (an empty string). With a reference table, you have greater flexibility when renaming or eliminating list choices (more on this below).

3. It's impossible to add additional attributes or related info.

There just isn't any sane way to add related information to an ENUM column, which is a common scenario that often comes up. In our country/continent example, what happens when we need to store something like land area for each continent? We didn't envision needing that attribute, but now we do. With a reference table, we can simply extend the continent table to include a 'land_area' column and query this new data any way we would like. With an ENUM? Forget it.

Other awesome flexibilities exist due to the ability to easily extend a reference table. One common scenario is adding a column to set a flag to denote whether a choice in the reference table is discontinued. So, when your company stops selling black widgets, you can add an 'is_discontinued' column to the reference table and flag the old 'black' row. You can still query a list of currently offered colors, and maintain info about all your old orders of black widgets! Try that with an ENUM column.

4. Getting a list of distinct ENUM members is a pain.

A very common need is to populate a select-box or drop down list with possible values from the database. Like this:

Select color:

redblueblack

If these values are stored in a reference table named 'colors', all you need is: SELECT * FROM colors ...which can then be parsed out to dynamically generate the drop down list. You can add or change the colors in the reference table, and your sexy order forms will automatically be updated. Awesome.

Now consider the evil ENUM: how do you extract the member list? You could query the ENUM column in your table for DISTINCT values but that will only return values that are actually used and present in the table, not necessarily all possible values. You can query INFORMATION_SCHEMA and parse them out of the query result with a scripting language, but that's unnecessarily complicated. In fact, I don't know of any elegant, purely SQL way to extract the member list of an ENUM column.

5. ENUM columns may only offer limited or negligible effects on optimization.

The usual justifications for using ENUM, are centered around optimization, in the conventional sense of performance gains, and sometimes in the sense of simplifying a complicated model to be more comprehensible.

Let's look at performance. You can do a surprising number of un-optimized things with databases, but most won't affect performance until a certain scale is reached, and often our applications are never asked to scale up that far. This is important to remember because DB devs should aspire to design fully-normalized and only de-normalize when a performance problem becomes real. If you're concerned that a reference table is going to slow things down, benchmark it out both ways in your unique application on an actual dataset (or a realistic high-estimate fake dataset) and see. Just don't automatically assume a join or a reference table is going to be a bottleneck, because it probably isn't. (There is also evidence to support that ENUM isn't always appreciably faster than alternatives.)

The second optimization argument for ENUM is that it reduces the number of tables and foreign keys in your database. This is a valid argument, in the sense that it's one more little box joined to another box with some lines, and in large systems the effect of normalization can already tax the limits of human comprehension and complicate queries. This is however, why we make models, and why those models employ abstraction so we can understand them. Go ahead and draw up a new representation of your model or ER diagram that leaves out some of the little details and reference tables. Sometimes it may just seem easier to use an ENUM, but the fact that you think another reference table makes things too complicated isn't a good reason by itself.

6. You can't reuse the member-list of an ENUM column in other tables.

When you create a list of possible members in an ENUM column, there's no easy and consistent way to re-use that list in other tables. With a reference table, the same set of data can be related to as many other tables as required. Changing the list in the lone reference table, will change the available options in every other table that it is linked or joined to.

With separate ENUM columns, you would have identical duplicate member lists on two different columns in at least two different tables (that would all require consistent updating).

7. ENUM columns have noteable gotchas.

Suppose you have ENUM('blue', 'black', 'red') and you attempt to insert 'purple': MySQL actually truncates the illegal value to  ' ' (an empty string).  This is correct, but if we had used a reference table with a foreign key, we would have more robust data integrity enforcement. 

Also, MySQL stores enum values internally as integer keys to reference ENUM members. It's easy to end up referencing the index instead of the value and vice-versa.  Consider:

CREATE TABLE test (foobar ENUM('0', '1', '2'));

mysql> INSERT INTO test VALUES ('1'), (1);
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

mysql> SELECT * FROM test;
+--------+
| foobar |
+--------+
| 1      |
| 0      |
+--------+
2 rows in set (0.00 sec)

We inserted '1' (a string), and accidentally also inserted 1 (as a number, without quotes).  MySQL correctly (but confusingly) uses our number input as an internal reference to the first item in the member list (which is actually the value '0').

8. ENUM has limited portability to other DBMS.

The ENUM data type isn't standard SQL, and beyond MySQL not many other DBMS's have native support for it. PostgreSQL, MariaDB, and Drizzle (the latter two are forks of MySQL anyway), are the only three that I know of. Should you or someone else want to move your database to another system, someone is going to have to add more steps to the migration procedure to deal with all of your clever ENUMs. If it's you, you'll undoubtedly feel less clever than you once did - and if it's someone else, they may not like you. Generally, migrating to a different database system is something that just doesn't happen that often and everybody assumes will bring out demons anyway, which is why this just squeaks in at number 8 on the list.

Criteria for when it might be okay to use enum:

1. When you're storing distinct, unchanging value sets...

A fairly good example that meets this criteria is our list of the continents. These are well-defined. Other commonly-given examples are salutations: Mr/Mrs/Ms, or playing card suits Spades/Hearts/Diamonds/Clubs. However, consider that even these examples have scenarios where you may need to extend the member list (such as when someone demands that you now need a 'Dr.' salutation, or when your card game app needs to accommodate a non-suited card like the Joker).

AND 2. You will never need to store additional related info...

Consider again Spades/Hearts/Diamonds/Clubs. There are popular card games that rely on the fact that clubs/spades are black and hearts/diamonds are red (Euchre, for example.) What happens when we need to store additional info related to suit, such as its color? If we had used a reference table, it would be a trivial matter to add this color data to the reference table in an additional column. If we use an ENUM to represent suit, it becomes much more difficult to represent the color/suit model accurately, and we're going to have to enforce it on the application level.

AND 3. The member list will contain more than 2 and less than 20 items.

If you're using an ENUM for only two values, you can always replace the ENUM with a very efficient TINYINT(1) or the even-better BIT(1) available since MySQL 5.0.3. For example: gender ENUM('male', 'female') can be changed to: is_male BIT(1). When you only have two choices, they can always be expressed as a Boolean true/false by prepending "is" to one of the member strings and renaming the column. As for less than 20: Yes, ENUM can store up to 65,535 values. No, you shouldn't try it. More than 20 becomes unwieldy and certainly more than 50 is just insane to manage and work with.

If you really still want to use ENUM:

1. Never use numbers as enum member values.

There's a reason ENUM is a string data type. Not only should you be using a numeric data type to store numbers, but ENUM has some well-documented gotchas related to the fact that  MySQL references ENUM members internally using a numerical index.  (See #7 above.)  Just don't ever store numbers in an ENUM data type, ok?

2. Consider using strict mode.

Strict mode will at least throw an error when you try to insert an invalid value into an ENUM column. Otherwise only a warning is thrown and the value is simply set to an empty string ' ' (referenced internally as 0). Note: Errors can still be suppressed in strict mode if you use IGNORE.

Conclusion

Do what makes sense from a development/maintenance perspective, and optimize only once a performance problem becomes real - in most cases that is a strong argument for using reference tables over MySQL's ENUM datatype.

There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.  - Donald Knuth

That works great for displaying special characters in the browser, but when populating a CKEditor instance with them, they are decoded back to the original character (and start causing all sorts of problems).

After hours of searching, I finally found this post on the CKEditor forums which at least described the problem:

...I think the actual problem is that CKEditor automatically converts bracket entities to actual brackets when initializing. So if the saved field HTML is:

<p>A paragraph starts with <code>&lt;p&gt;</code>.</p>

CKEditor will automatically convert the `&lt;` and `&gt;` entities to `<` and `>` while initializing, and then parse it as an actual `<p>` tag.

This is exactly what I observed, and no combination of configuration options in the CKEditor API seems capable of disabling this behavior.  (Believe me, I tried, and even hacked the source for a few hours in vain.)  In my case, I am pulling html-formatted text from a database (which frequently contains code snippets with encoded < symbols), editing in CKEditor, and inserting back into the database.  How can I preserve a string like <?php when it enters CKEditor, so that it displays properly and can be edited and re-saved? 

The Solution

Double encode all entities found in the source data before CKEditor initializes.    Specifically what this means is to convert all occurrences of the ampersand character, &, to & before CKEditor gets its hands on the data.  With PHP:

// before being fed to the textarea of CKEditor
$content = str_replace('&', '&', $content);

You may be thinking: "doesn't PHP have htmlspecialchars() and/or htmlentities() functions that could be called instead of a string replacement?"  The answer is yes, but these functions will encode much more than what we need here (thus making CKEditor's source view look like a mess of entities instead of html with a few special entities inserted where needed).  We don't need every < > " ' & converted to an entity, in fact we only need '&' converted to '&'.

The flow here looks something like this for various character sequences:

The critical things to note here are:

1. The literal < and > characters maintain state throughout, as they should - you want your real <p> and other tags to stay real tags the whole way through.
2. To display the literal < or > or & strings on screen, they will have to be stored double-encoded in the database or file (and therefor triple-encoded like & when sent to CKEditor).
3. This method requires that the following CKEditor configuration options be used:

   config.entities = false;
   config.htmlEncodeOutput = false;
   

4. CKEditor essentially strips off the double-encoding when it loads, so there is no need to perform any decoding or special replacements with the data submitted by the editor - it will be saved correctly in the database or file.

As this post (with a ton of entities needed to display properly) was created and edited in a system which uses the technique, I'd say it works!

There are no new ideas. There are only new ways of making them felt. - Audre Lorde

I've spent a significant portion of my life thinking that someday, I'd have some great idea that would bring me some combination of fame, fortune, or fulfillment. That there was some truly great as-yet-un-thought thing that could stand on its own for being brilliant. This idea would be so revolutionary and unique that it would rule whatever market or arena it fell into without much effort at all.

In the real world, this just doesn't happen. For starters, history is littered with examples of great inventors and business people who are probably mistakenly given all the credit for a particular idea. Alexander Bell. The Wright Brothers. Henry Ford. Thomas Edison.

Edison held over 1000 patents in the US alone and is commonly believed to have invented everything from the light bulb to the stock ticker. In reality most of those patents are for improvements to prior art, and a little digging reveals an interesting drama between Edison, Westinghouse, Tesla, and lesser-known contemporaries where it becomes almost impossible to say who came up with what, or when. Often it seems, what matters is not the idea, but how it was marketed (in a very general sense of the word). There was this thing called electricity and it was big and it really belonged to no one.

Fast forward to today's electrical frontier. Microsoft didn't invent windowed application frames any more than Google invented the search engine, Oracle invented the database, or Quora invented people asking and answering questions on the internet. There was this thing called the information age and it was big and it really belonged to no one.

Joel Spolsky says this best in Converting Capital Into Software That Works:

The common belief is that when you're building a software company, the goal is to find a neat idea that solves some problem which hasn't been solved before, implement it, and make a fortune. We'll call this the build-a-better-mousetrap belief.

First of all, many of the most successful software companies (Microsoft and Oracle, for example) don't really "innovate" in the sense that they are not really solving problems that haven't been solved before. In any market, it is exceedingly rare that you get to keep your invention to yourself. Everybody has competition.

People get wrapped up in protecting and patenting their ideas to the point of paranoia. I've witnessed companies that were so bad they were afraid to effectively market their product because the competition would know about it. Neither did their customers of course, but that didn't seem to matter because their focus was on the idea, not "how it was felt".

Stop worrying. If you build it, they will come. If it's good, they will come a lot and pass by your competition to get there. Stop treating your ideas like some sort of cloaked-page keyword-stuffed SEO tactic, where you're going to somehow game the system and beat everyone to number one on Google. People thought of that, just like people probably already thought of your idea, and it didn't work. Making something that people feel differently or better about is more important than protecting ideas that you haven't even implemented well -- just like the most successful strategy for SEO is having good content that people value and not worrying about anything else.

In How to Start a Startup, Paul Graham writes:

Ideas for startups are worth something, certainly, but the trouble is, they're not transferable. They're not something you could hand to someone else to execute. Their value is mainly as starting points: as questions for the people who had them to continue thinking about.

What matters is not ideas, but the people who have them. Good people can fix bad ideas, but good ideas can't save bad people.

Nick Cash also quoted this in a similar post in response to Eugene Wallingford's post: I just need a programmer. Nick goes on to say: "This brings up the most fundamental point in business, and probably life itself: its all about the people."

I've worked for some awesome companies that made good people great, and I've worked for some absolute disasters where managers thought motivation was best supplied by posting a job ad for your position on the internet. I always joke that someday I'm going to start a company named "We're Not Idiots". Aside from the advertising copy basically writing itself, it doesn't seem a bad premise for a business. It does have a ring of Google's "Don't Be Evil" about it. I'm continually amazed at how many companies (with otherwise workable ideas) just don't get it. Can't anybody play this game?

Find good people and treat them with respect. Make it your mission to enable them. The people around you will elevate or cripple you, and they'll certainly elevate or cripple your ideas.

What you make or do is secondary. The ideas you have are secondary. And your ego comes dead last.

--

Original image credit: _Yoshi_

Most things don't work yet. Comments are working. Happy holidays!

Update: the old RSS feed is, (and will remain) working.

The concern with smart phones has always been "Yeah, ok it can do that, but is the feature actually usable"? There have been a lot seemingly-useful capabilities that users rarely touched after the initial "wow, cool" gimmick period expired. Mostly, the culprit tends to either be interface or speed, and Apple's pretty slick interface coupled with 3g connectivity (and prolific wifi) seems to satisfy: we actually want to use the features.

There are still some painful (and well documented) omissions (MMS anyone?). Apple's policies regarding the flexibility of the device do indeed suck. The wonderful app store which you see advertised on TV every 5 minutes may have 30,000+ apps, but 29,900 of them are rather pointless. Jailbreaking the device, fortunately, is a painless 10-minute process which satisfyingly allows a curious user to correct most of these issues.

One great actively-developed app (only available on jailbroken devices) is xGPS, which "aims to bring a powerful and easy to use navigation software for the Apple iPhone and iPod Touch." We recently put it through it's paces on an impromptu late evening off-road foray, far from the reaches of five-bar cellular signals- a trip which would not have even been attempted had the iPhone not been with us.

Fortunately, we had preloaded the phone with about 60MB of map data (an extremely valuable feature of xGPS). For a good portion of the trip there was no cellular signal available-- without preloaded map data, we would have only been able to know our latitude and longitude and would have little idea where we were, relatively speaking. Other valuable features we discovered included "night mode", which was wonderfully easy-on-the-eyes after dark on narrow trails which required the driver's attention, and the availability of terrain maps with shading and contour lines to give you the lay of the land.

While we're no Apple fan-boys and we enjoy disconnecting from the technological world as often as possible, having this kind of actually usable technology at-hand, makes us a little giddy. With a calendar, a todo list, email, instant messaging, a personal music collection, Pandora radio, a camera, a web browser, daily newspapers, and more all in your pocket on one slim device, it's hard not to feel like some part of the futuristic world we were promised years ago has been realized.

(Side note: at present, it appears that any one of several tab-related Firefox add-ons can be used to improve the "new tab" functionality. Our pick: New Tab Homepage)

Most promising of the Mozilla Labs projects may be Prism (formerly named WebRunner):

Prism is a simple XULRunner based browser that hosts web applications without the normal web browser user interface. Prism is based on a concept called Site Specific Browsers (SSB). An SSB is an application with an embedded browser designed to work exclusively with a single web application. It doesn't have the menus, toolbars and accoutrements of a normal web browser. Some people have called it a "distraction free browser" because none of the typical browser chrome is used. An SSB also has a tighter integration with the OS and desktop than a typical web application running through a web browser.

Not so long ago, we wrote about a minor epiphany of sorts where we "discovered" browser-based programming. Since then, we've been fortunate to become gainfully employed in developing browser-based MIS applications. Our discovery of and increasing creativity with the jQuery javascript library and Ajax, has made available many interface options previously unavailable to traditional web-apps.

It consistently amazes us how we can be mostly unaware of whole concepts, such as the SSB (Wikipedia), while having problems that cry out for the very solutions these concepts provide. In truth, we have long thought about about SSB's, without knowing the term... at one point even working on a C# .NET-based Windows GUI that was in hindsight, exactly a SSB: nothing more than a Windows WebBrowser control with a custom lightweight chrome. Without really knowing what we were doing, we were trying to bridge the interface gap between the web and the desktop.

Prism (and any of the other SSB packages out there) may very well address what is undoubtedly the largest headache for web application developers: deploying in uncontrolled and varied environments. Much has improved with regards to browser and platform inconsistencies in the last five years, but on closer inspection, many of these "improvements" are merely elegant hacks that developers have used to abstract away from the fact that no two browsers (or rendering engines) are likely to ever be created equal.

There is, in all likelihood, a distinction that needs to be made here between a "web designer" and a "web-application developer". The web designer, primarily tasked with presenting and formatting content, may be forced to (and ethically should) continue developing for the range of browsers in the uncontrolled environment of the Internet. The web app developer on the other hand, is tasked with creating much more interface-heavy applications that would benefit from (and increasingly may demand) a more controlled platform/environment for deployment... which Site Specific Browser packages like Prism may well elegantly provide.

php_flag display_errors on
php_value error_reporting 6135

We initially thought that perhaps we could get away with using PHP's defined constants (such as E_ALL) directly in the .htaccess file, but unfortunately .htaccess does not appear to have access to such PHP constants, and integer values must be used to set the error reporting level.

There does seem to be some confusion out there as to how these integer values are calculated, although it appears quite simple. The values are arrived at by either adding or subtracting component values, as declared in the PHP manual.

For example, E_ALL's value is 6143, and E_NOTICE's value is 8. So, by subtracting 8 from 6143, we arrive at 6135, which as noted above is equivalent to "E_ALL & ~E_NOTICE", or something like "all errors, except notices". You can always check the integer values for a given error-reporting level:

<?php
ini_set("error_reporting", E_YOUR_ERROR_LEVEL);
echo ini_get("error_reporting");
?>

Of course, you can substitute any error reporting values. For maximum error reporting and warnings (as recommended for any PHP5+ dev environment):

php_flag display_errors on
php_value error_reporting 8191

*You should note that these integer values may produce unexpected results in future versions of PHP and are not guaranteed to be forward compatible. From the PHP site:

Using named constants is strongly encouraged to ensure compatibility for future versions. As error levels are added, the range of integers increases, so older integer-based error levels will not always behave as expected.

...which of course is problematic, as we cannot use named constants in .htaccess. So, the caveat here is that this per-directory .htaccess error level setting may need updated if/when you upgrade PHP versions. (PHP 6 rolls E_STRICT into E_ALL, so the recommended practice to ensure maximum error reporting "from now and well into the future" is to simply use a large "numeric value like 2147483647".)