The server in question was aramis.rutgers.edu. It was listed in the netgroup as aramis.rutgers.edu. A DNS lookup of it's IP, 22.214.171.124, returned aramis.rutgers.edu. Everything worked fine.
Then one day, DNS lookups for 126.96.36.199, started returning aramis.Rutgers.EDU. Shortly thereafter, mounts on aramis from the filer started failing. Exports to some critical filesystems on the filer were fixed immediately by adding aramis.rutgers.edu explicitly to the filer's exports file. After lots of consultation with NetApp, we determined that the access via the netgroup was failing because the case of the hostname in the netgroup did not match the case returned from DNS. While they agreed that hostnames in DNS were case insensitive, they maintained that the case of the results returned from a DNS query must match what is in the netgroup. That is, hostnames (and anything else) obtained from netgroups are case-sensitive. They pointed to
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-nis.htmlto show that hostnames in netgroups under FreeBSD (and HP) are case-sensitive too. Their suggested solution was to change either DNS or the /etc/netgroup file so that the cases in both match. So I reluctantly changed the case in the netgroups and wrote a script to watch for further case changes in hostnames we were using in netgroups. Immediate problem solved. But should the case change in the result of some other IP in a netgroup used on the filer the problem would return, sort of like a game of Whac-A-Mole.
In the meantime, I contacted our network group to determine why the case had suddenly changed in DNS, since the database from which the DNS data were generated had it properly in lowercase. After some investigation, they reported that the change of case is coming in through the zone transfer process. They admitted their current infrastructure does not fully comply with RFC 1034, but pointed out that the RFC says "By convention, domain names can be stored with arbitrary case," but "when you receive a domain name or label, you should preserve its case."
So the problem was in zone transfers... At this point, I wrote a wrapper script to lowercase all occurrences of ".rutgers.edu." for zone transfers done by named-xfer on our locally run DNS server. Once the next transfer of our IN-ADDR.ARPA zone took place, aramis.rutgers.edu was properly lowercased in lookups using that server. At that point, I lowercased aramis.rutgers.edu in the netgroups on the filer, and all worked fine again.
End of story? Not quite. /etc/netgroup on many of our machines is built from a single configuration file. So now a number of our machines now had the mixed case aramis.Rutgers.EDU in their netgroups. Once aramis.rutgers.edu was all lowercase in DNS, mounts on aramis from a different Solaris server (which still had the mixed case aramis.Rutgers.EDU in its netgroups) started failing. Apparently, Solaris is willing to lowercase results from DNS to compare a hostname with one contained in a netgroup, but respects mixed case found in a netgroup and does not do the comparison case-insensitively. The mount continued to fail until the hostname was lowercased in that server's netgroup.
During this ordeal, which lasted over two weeks start to finish, I did several web searches about case sensitivity of hostnames in netgroups and found nothing which explained well what I was seeing. I put this out in the hope of saving someone else the trouble I went through.