What is the HOSTNAME= value set to in /opt/scalix/global/config? If it is just the host name try changing it to the fqdn. Restart tomcat.
Hi. One other thing, the reason mail is piling up in /var/spool/mqueue (I'm assuming you meant mqueue rather than mail) is because you don't have your domain/server in the /etc/mail/local-host-names file. Update that file, then restart sendmail and those messages will bounce.
It used to be the case that MaxMessageSize in /etc/mail/sendmail.cf was set to 10M but, looking at my file, this is commented out and the default is 0, i.e. unlimited.
omon -s all
1005 vi /etc/mail/local-host-names
# exactly -- but if they are a little bit late, the server drops their
# connection. Setting a timeout of 31 minutes avoids this problem.