[... This post got too long for no good reason, feel free to jump to the conclusions ...]

The problem
On my CentOS 5 server, the default sound device (/dev/snd/*) permission was root:root 0600.
This means that other users simply cannot play music. Annoying.

The solution ought to be simple...
So, I've ran groupadd sound and added all the relevant users to the new group.

Then, I went to /etc/udev/rules.d/90-alsa.rules, and added this line, to tell that anything that is sound related, should be fully accessible to "sound" group:

SUBSYSTEM=="sound", GROUP="sound", MODE="0660"

Fanatic problem solving mode: ON
But.. not working. Then I've switched into "fanatic problem solving" mode. This means: trying everything without too much thought, modifying every possible file with any possible way.. But no luck. /dev/snd/* files are still root-only-accessible.

A-HA!
Then I've tried to really think.. Running the following command and really inspect its output:

udevtest /devices/audio/subsystem/timer

(udevtest needs the /sys path and not /dev path, quite annoying)

The output shows the list of rules that udev would run for this device. Then I've noticed the last line

main: run: '/sbin/pam_console_apply /dev/snd/timer '

PAM! Of course it's guilty. Then a quick grep revealed the following in /etc/security/console.perms.d/50-default.perms:

<console> 0600 <sound> 0600 root

I commented it out, and ... viola! All works.

---

In short...

Linux, or at least RedHat 5, has two conflicting mechanisms for setting /dev file permissions:
1. udev: the service responsible for /dev directory content.
2. pam: the service (well, not quite a service but something similar) responsible for system's security.

Udev actually calls the pam service on new device (on rules.d/95-pam-console.rules), so in effect, pam might actually override Udev permission settings. The relevant PAM config files are in /etc/security/console.perms.d).

Great, 1.5hrs got wasted. at least I've learned something.