Ticket #1192 (closed defect: fixed)

Opened 11 years ago

Last modified 10 years ago

unknown reason cause TI GSM modem always return "+CME ERROR: 512"

Reported by: sean_chiang@… Owned by: openmoko-devel
Priority: high Milestone:
Component: hardware Version:
Severity: critical Keywords:
Cc: testing@…, mickey@…, buglog@…, erin_yueh@…, chris@…, joerg@… Blocked By:
Blocking: Estimated Completion (week):
HasPatchForReview: no PatchReviewResult:
Reproducible:

Description

For now, I have no idea what makes TI GSM modem return "+CME ERROR: 512". But
when modem enter this state, no mater what atcommand we send to modem, we always
get "+CME ERROR: 512". Hence, we couldn't make phone call anymore.

Attachments

gsm-0.log (116.5 KB) - added by sean_chiang@… 11 years ago.
gsm.log
gsm-2.log (112.3 KB) - added by sean_chiang@… 11 years ago.
another gsm log from neo(gta01)
02-request-imsi-when-init-gsmd.patch (1.5 KB) - added by sean_chiang@… 11 years ago.
retrieve imsi when init gsm
logread2.txt (63.2 KB) - added by octavsly 10 years ago.
Attachements of ticket 1738 (requested to be moved here)
qtopia-restart-the-gsm-modem-on-firmware-crash.patch (4.1 KB) - added by Treviño 10 years ago.
Restart the GSM modem on firmware crash
The firmware of modem appears to have crashed (62.6 KB) - added by wendy_hung 10 years ago.

Change History

comment:1 Changed 11 years ago by sean_chiang@…

  • Status changed from new to assigned

Changed 11 years ago by sean_chiang@…

gsm.log

Changed 11 years ago by sean_chiang@…

another gsm log from neo(gta01)

Changed 11 years ago by sean_chiang@…

retrieve imsi when init gsm

comment:2 Changed 11 years ago by sean_chiang@…

  • Status changed from assigned to closed
  • Resolution set to fixed

Fixed it

comment:3 Changed 11 years ago by sean_chiang@…

  • Status changed from closed to reopened
  • Resolution fixed deleted

It still happen, but now I could re-produce it.

  1. Make our test environment simple

stop xserver, and kill phone-kit process


  1. camp to network first through libgsmd-tool


  1. Try this command 2 times, then I got "+CME ERROR 512" at second time.


libgsmd-tool -m shell < test
The content of test is:


n


S=5
D0937xxxxxx
S=10
H
S=5


  1. After we got "+CME ERROR 512", TI GSM modem is out of control.


Whatever atcommand(ex. AT) we send, it always respond "+CME ERROR 512"

comment:4 Changed 11 years ago by sean_chiang@…

  • Cc mickey@… added

comment:5 Changed 11 years ago by sean_chiang@…

  • Cc chris@… added

comment:6 Changed 11 years ago by erin_yueh@…

i tried to reproduce this CME error 512 with your steps, but i didn't get this
error. -erin

comment:7 Changed 11 years ago by erin_yueh@…

but i got cme error 512 from #1195 cnum test case.

comment:8 Changed 11 years ago by erin_yueh@…

oops...... i wrote a wrong test case, and i forgot the first 'n' command.
after using this test case, i got that magic error code too!

comment:9 Changed 11 years ago by balrogg@…

Hi,
probably not very helpful, but I remember my Calypso modem on GTA01 sometimes
getting into a loop where it would only give a +CME error on anything I do until
I reset it. It was not reproducible - when it happened for the first time, I
tried to reproduce it and couldn't, but another day ran into it again. It was a
long time ago.

I don't know what the error code was because I was using the AT+CMEE=2 mode
(textual errors) so it was something like:

+CME: Operation in progress

or something similar, but it was a non-standard error (i.e. it was not in the
GSM specs when I checked). I only hit it when manually using the serial port
with "cu".

The modem was printing this error on every *character* I typed, not every
command (unless pasting commands). So if I typed AT+ABC slowly, I would get:

A
+CME: Operation in progress
T
+CME: Operation in progress
+
+CME: Operation in progress
A
+CME: Operation in progress
B
+CME: Operation in progress
C
+CME: Operation in progress

comment:10 Changed 11 years ago by balrogg@…

So I think it's not a gsmd bug.

comment:11 Changed 11 years ago by sean_chiang@…

Yes, not gsmd problem, the problem definitely is coming from GSM firmware. If we
remove the "AT+COLP=1" in gsmd_initsettings2(), then it won't happen anymore.

comment:12 Changed 11 years ago by mickey@…

Is this really the latest firmware from TI? I have the very uncomfortable
feeling that some of our problems are not due to kernel serial port or gsmd but
actually due to the firmware. Can we double check that with TI?

comment:13 Changed 11 years ago by sean_chiang@…

It's the latest TI GSM firmware from TI, if "latest" means from TI. Because we
had already do some modification since the lastest firmware delievered from TI.
Before solve this from firmware side, we'll have a work-around from gsmd side.

comment:14 Changed 11 years ago by mickey@…

  • Status changed from reopened to closed
  • Resolution set to wontfix

Ok, I just managed to get to this state again. I did multiple sequences of
at+cops=? followed by cancelling (sending any other character) the command. It
seems the AT interpreter or firmware state machine inside the modem just hangs
somewhere undefined.

Additional remarks:

  • There does not seem to be a concrete way to trigger this state. It seems

unrelated to any sequence of commands. It's just that commands with long
execution time seem to trigger it more often.

  • It also happens in MUXing mode and it only renders one channel unusable. The

other channels still work as expected.

  • There does not seem a way to recover from this state. Even closing and

reopening the virtual channel does not fix the problem. Even waiting one hour
does not fix the problem. The only way to recover is doing a modem power cycle.

Since the TI Calypso is end-of-life it's highly unlikely that someone will look
into this problem. Since it occurs very rare and also can be detected by a good
parser (which could then transparently trigger the modem power cycle), I'll
close this as WONTFIX.

comment:15 Changed 11 years ago by zecke

  • Status changed from closed to reopened
  • Component changed from gsmd to Qtopia
  • Resolution wontfix deleted

I fear power cycling is not a good solution.

comment:16 Changed 11 years ago by roh

  • Owner changed from sean_chiang@… to sean_chiang

comment:17 Changed 10 years ago by octavsly

I see the same problem here. When, I could not get the call I had:
Sep 3 15:16:17 om-gta02 user.notice root: AtChat? : F : "+CMS ERROR: 512"
Sep 3 15:16:17 om-gta02 user.notice root: AtChat? : T : "AT+CFUN=1"
Sep 3 15:16:17 om-gta02 user.notice root: AtChat? : F : "+CMS ERROR: 512"
Sep 3 15:16:17 om-gta02 user.notice root: AtChat? : T : "AT+CFUN?"
Sep 3 15:16:18 om-gta02 user.notice root: AtChat? : F : "+CMS ERROR: 512"
Sep 3 15:16:18 om-gta02 user.notice root: AtChat? : T : "AT+CMEE=1"

After this the phone keeps on waking up as mention in #1875.
The message this time is:
Sep 3 15:25:19 om-gta02 user.notice root: Modem : void CellModemManager::retryRfLevelRequest() Retrying cfun
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : F : "ERROR"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : T : "AT+CFUN=1"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : F : "ERROR"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : T : "AT+CFUN?"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : F : "ERROR"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : T : "AT+CMEE=1"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : F : "ERROR"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : T : "AT+CFUN=1"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : F : "ERROR"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : T : "AT+CFUN?"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : F : "ERROR"
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : T : "AT+CMEE=1"
Sep 3 15:25:19 om-gta02 user.notice root: Modem : void CellModemManager::retryRfLevelRequest() Retrying cfun
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : F : "ERROR"
Sep 3 15:25:19 om-gta02 user.notice root: Modem : void CellModemManager::retryRfLevelRequest() Retrying cfun
Sep 3 15:25:19 om-gta02 user.notice root: AtChat? : T : "AT+CFUN=1"

Changed 10 years ago by octavsly

Attachements of ticket 1738 (requested to be moved here)

Changed 10 years ago by Treviño

Restart the GSM modem on firmware crash

comment:18 Changed 10 years ago by Treviño

Hello, I've tried to improve the management of this issue with the patch above. Since all the times I got this crash I was able to recover my phone simply restarting the GSM interface with:

killall qpe
echo 0 > /sys/devices/platform/neo1973-pm-gsm.0/power_on
echo 1 > /sys/devices/platform/neo1973-pm-gsm.0/power_on
qpe

I've tried to get the same from the modem plugin. Now qtopia should restart the gsm interface and then reconfigure the plugin as on first run.

Unfortunately I couldn't try this (since fortunately I had no crash in the past days :D), but I hope it will work.

comment:19 Changed 10 years ago by wendy_hung

  • Owner changed from sean_chiang to openmoko-devel
  • Status changed from reopened to assigned
  • HasPatchForReview unset
  • Version 2007.2 deleted

Don't know if it helps, but here is the logread that i happened with this problem. block with #1738.
use the testing branch image flash at 10/28, opkg update/upgrade at 10/29

Changed 10 years ago by wendy_hung

comment:20 Changed 10 years ago by wendy_hung

  • Cc testing@… added

comment:21 Changed 10 years ago by mickey

Fwiw, I learned from the GSM firmware that this error means "unable to cancel command". The gsm server should be very careful not to try cancelling a command twice. Also, some commands should not be cancelled (i.e. +COPS=?). Cancelling an ATD seems to work fine, so every gsm server using the MUX mode should be able to get away without provoking this illegal state -- or at least transparently power-cycle the modem after you encounter this message. The core problem is definitely in the firmware, but it's unlikely we will see a fix for that.

comment:22 Changed 10 years ago by mickey

  • Component changed from Qtopia to hardware

I just got the first report from someone whose 512 was fixed by updating to moko10-beta2 firmware. Although this may be unrelated (since we still don't know which exact conditions lead to 512), it may be worthwhile to try. 512 may be connected to Om ticket #666.

comment:23 Changed 10 years ago by mickey

Someone also needs to double check whether http://docs.openmoko.org/trac/changeset/3964 has any effect on 512 or not.

comment:24 Changed 10 years ago by joerg

  • Cc joerg@… added

added to CC

comment:25 Changed 10 years ago by mickey

Seems pretty much contained with disabling +COLP. Has anyone seen it _with_ +COLP=0?

comment:26 Changed 10 years ago by mickey

  • Status changed from assigned to closed
  • Resolution set to fixed

+COLP=0 and moko11 fixes this for good.

Note: See TracTickets for help on using tickets.