Operating system name and version: CentOS 5.5
Version of smsd: v3.1.14
Smsd installed from: sources
Name and model of a modem / phone: Modem WaveCom 900E
Interface: serial on COM1

Our initial problem was that the modem we use, hangs every once and a while.
If this happens then we get log entries such as:
--------------
2011-06-25 11:07:03.018,2, smsd: Smsd v3.1.14 started.
2011-06-25 11:07:03.018,2, smsd: Running as root:root.
2011-06-25 11:07:03.020,7, smsd: Running startup_check (shell): /var/spool/sms/incoming/smsd_script.uB3k9J /tmp/smsd_data.3ezypi
2011-06-25 11:07:03.031,7, smsd: Done: startup_check (shell), execution time 0 sec., status: 0 (0)
2011-06-25 11:07:03.032,4, smsd: File mode creation mask: 022 (0644, rw-r--r--).
2011-06-25 11:07:03.032,4, smsd: Mainspooler uses GSM1 to send administrative messages.
2011-06-25 11:07:03.032,6, smsd: Using logtime format "%Y-%m-%d %H:%M:%S.timems". It produces "2011-06-25 11:07:03.032".
2011-06-25 11:07:03.032,5, GSM1: Modem handler 0 has started. PID: 28533.
2011-06-25 11:07:03.032,5, GSM1: Using check_memory_method 1: CPMS is used.
2011-06-25 11:07:03.040,5, smsd: Outgoing file checker has started. PID: 28532.
2011-06-25 11:07:03.040,7, smsd: All PID's: 28532,28533
2011-06-25 11:07:03.056,6, GSM1: I have to send 1 short message for /var/spool/sms/checked/247751308772811
2011-06-25 11:07:03.056,6, GSM1: Checking device for incoming SMS
2011-06-25 11:07:03.056,6, GSM1: Checking if modem is ready
2011-06-25 11:07:03.257,7, GSM1: -> AT
2011-06-25 11:07:03.258,7, GSM1: Command is sent, waiting for the answer
2011-06-25 11:07:03.560,7, GSM1: <- OK
2011-06-25 11:07:03.560,6, GSM1: Pre-initializing modem
2011-06-25 11:07:03.762,7, GSM1: -> ATE0+CMEE=1;+CREG=2
2011-06-25 11:07:03.764,7, GSM1: Command is sent, waiting for the answer
2011-06-25 11:07:04.374,7, GSM1: <- OK +CREG: 0
2011-06-25 11:07:04.374,6, GSM1: Checking if modem needs PIN
2011-06-25 11:07:04.590,7, GSM1: -> AT+CPIN?
2011-06-25 11:07:04.591,7, GSM1: Command is sent, waiting for the answer
2011-06-25 11:07:04.898,7, GSM1: <- +CME ERROR: 13 (SIM failure)
2011-06-25 11:07:04.898,2, GSM1: PIN handling: expected READY, modem answered +CME ERROR: 13 (SIM failure), SIM failure
2011-06-25 11:07:04.898,2, GSM1: Modem handler 0 terminated abnormally. PID: 28533.
--------------

After that, the main process is still running but as there is no modem handler any more, the whole system
does not work again.

We tested different scenarios and came to the conclusion that power OFF/ON resets the modem to a defined state.
In order to power cycle the modem we have installed a "remote power switch" which we can program via curl-commands.

During testing we came across the situation that smsd has sent a command and waited for an answer.
As we have powered off the unit it never came back.

we've got the following entries in our logfile - but only after we have terminated the smsd-jobs via /etc/init.d/smsd stop


--------------
2011-06-27 08:45:38.584,6, GSM1: Checking if modem needs PIN
2011-06-27 08:45:38.786,7, GSM1: -> AT+CPIN?
2011-06-27 08:45:38.786,7, GSM1: Command is sent, waiting for the answer
2011-06-27 08:45:39.090,7, GSM1: <- +CME ERROR: 515 (device busy)
2011-06-27 08:45:39.090,2, GSM1: PIN handling: expected READY, modem answered +CME ERROR: 515 (device busy), device busy
--------------


The question now is, if there is any parameter that we can use in order to avoid a blocking smsd due to a unit that does not answer any more.

Right now we have to run a cronjob which checks for the number of smsd-processes and if this is < 2, we run our script to
first power off the unit, run the stop-script, power on the unit and start the start-script.

Hope you can give us some help on how to make this easier.

Kind regards
Thomas Thim

Basically your solution is good, but the modem is bad as it does not always work properly. Have you checked if the power supply is weak, or noisy?

In the script sms3 there is a definition RESETMODEMS=/usr/local/bin/smsd_resetmodems. The script sms3 can be called with argument reset, which causes that smsd is stopped, reset is done, and smsd is restarted. Your cron job can do that.

Handling of PIN is a special case: if the handling fails, modem process will stop. If some error happens when smsd is trying to communicate with modem, alarmhandler is called. You could define alarmhandler, and create a log by it. From the log it can be seen what kind of failures happens, and decide what kind of failures should cause restarting. For example see this post. There is a "controlfile" created, and external process can find it and perform the reset.