Ticket #1728 (closed defect: fixed)

Opened 11 years ago

Last modified 10 years ago

SMS parser bug?

Reported by: odlg Owned by: tick
Priority: high Milestone:
Component: Qtopia Version: Om2008.8
Severity: major Keywords:
Cc: mickeyl@…, tick@… Blocked By:
Blocking: Estimated Completion (week):
HasPatchForReview: PatchReviewResult:
Reproducible:

Description

Google calendar has a feature to send SMS's before some event, I use that a lot. The problem is that the SMS's from Google are not readable on ASU. It should tell something like "Firma møde kl. 8.30"

Notice on the "screenshot" that there is some garbage after Google as sender, the SMS's does not include sender phone number, just "Google" - does the SMS parser not support that?

Attachments

dsc_5992s.jpg (89.3 KB) - added by odlg 11 years ago.
Screenshot of unreadable SMS
Screenshot-1.png (15.8 KB) - added by odlg 11 years ago.
Screenshot-2.png (40.0 KB) - added by odlg 11 years ago.
Screenshot-24.png (60.8 KB) - added by Treviño 10 years ago.
Badly decoded SMS

Change History

Changed 11 years ago by odlg

Screenshot of unreadable SMS

comment:1 Changed 11 years ago by zecke

  • Cc mickeyl@… added

Could you please get me the PDU of that message? Please see #1649 on a small note how to get the PDU for me.

comment:2 Changed 11 years ago by odlg

So far I only get this:

Aug  2 22:59:19 om-gta02 user.notice root: AtChat :  T : "AT+CMGF=0" 
Aug  2 22:59:19 om-gta02 user.notice root: AtChat :  F : "OK" 
Aug  2 22:59:19 om-gta02 user.notice root: AtChat :  T : "AT+CPMS="SM"" 
Aug  2 22:59:19 om-gta02 user.notice root: AtChat :  F : "+CPMS: 9,200,9,200,9,200" 
Aug  2 22:59:19 om-gta02 user.notice root: AtChat :  F : "OK" 
Aug  2 22:59:19 om-gta02 user.notice root: AtChat :  T : "AT+CNMI=2,1,2,0,0" 
Aug  2 22:59:19 om-gta02 user.notice root: AtChat :  F : "OK" 
Aug  2 22:59:19 om-gta02 user.notice root: AtChat :  T : "AT+CMGL=4" 
Aug  2 23:00:25 om-gta02 user.err kernel: power_supply bat: driver failed to report `capacity' property
Aug  2 23:03:08 om-gta02 user.err kernel: power_supply bat: driver failed to report `time_to_full_now' property
Aug  2 23:03:09 om-gta02 user.err kernel: power_supply bat: driver failed to report `capacity' property
Aug  2 23:03:17 om-gta02 user.notice root: "qtmail" raised 
Aug  2 23:03:50 om-gta02 user.err kernel: power_supply bat: driver failed to report `current_now' property
Aug  2 23:05:11 om-gta02 user.notice root: AtChat :  N : "%CSQ:  16, 99, 1" 
Aug  2 23:05:11 om-gta02 user.notice root: AtChat :  N : "+CIEV: 1, 2" 
Aug  2 23:05:18 om-gta02 user.notice root: AtChat :  N : "%CSQ:  20, 99, 2" 
Aug  2 23:05:19 om-gta02 user.notice root: AtChat :  N : "+CIEV: 1, 3" 
Aug  2 23:05:25 om-gta02 user.err kernel: power_supply bat: driver failed to report `time_to_full_now' property
Aug  2 23:05:29 om-gta02 user.notice root: Modem :  QModemCall::hangup() 
Aug  2 23:05:29 om-gta02 user.notice root: Modem :  hangup groups 

I am unable to make calls. :-( I will reboot the phone and try again.

comment:3 Changed 11 years ago by odlg

OK, here are a couple:

Aug  2 23:13:46 om-gta02 user.notice root: AtChat :  F : "+CMGL: 7,1,,84" 
Aug  2 23:13:46 om-gta02 user.notice root: AtChat :  F : "07918167830071F1040BD0C7F7FBCC2E030000808010800120804AD0473BED2697D9F3B20E644CCBDBE136835C6681CCF2B20B147381C2F5B30B04C3E96630500B1483E96030501A34CDB7C5E9B71B847AB2CB2062987D0E87E5E414" 
Aug  2 23:13:46 om-gta02 user.notice root: AtChat :  F : "+CMGL: 10,1,,124" 
Aug  2 23:13:46 om-gta02 user.notice root: AtChat :  F : "07918167830071F1040BD0C7F7FBCC2E0300008080203200748078D0473BED2697D9F3B20E442DCFE9A076793E0F9FCBA07B9A8E0691C3EEF41C0D1AA3C3F2F0985E96CF75A00EE301E22C1C2C109B217781642E50B87E76816433DD0C066A81E60CB70B347381C2F5B30B

comment:4 Changed 11 years ago by odlg

Hmm, according to http://twit88.com/home/utility/sms-pdu-encode-decode something is wrong with the second PDU, here I try again:

07918167830071F1040BD0C7F7FBCC2E0300008080203201208078D0473BED2697D9F3B20E442DCFE9A076793E0F9FCBA07B9A8E0691C3EEF41C0D1AA3C3F2F0985E96CF75A00EE301E22C1C2C109B217781642E50B87E76816433DD0C066A81E60CB70B347381C2F5B30B

comment:5 Changed 11 years ago by zecke

thanks

comment:6 Changed 11 years ago by Ebbe

Just to let you know: FSO milestone 2 displays those correctly.

comment:7 Changed 11 years ago by avanc

It seems that Qtopia can't handle SMS where the "Sender" field contains letters. I sent two messages via the Sipgate webinterface. One with a sender number and one without (Sipgate sets "sipgate" as sender by default).

Worked:

0791447758100650040C9194714373238200008080312160304019D4F29C0E6A97E7F3F0B90CB2A7C3A0791A7E0ED3CB2E

Didn't worked:

0791447758100650040DD0F334FC1CA6970100008080312170224008D4F29CDE0EA7D9

comment:8 Changed 11 years ago by zecke

  • Status changed from new to accepted
  • Version changed from GTA02v5 to OM-2008.08
  • Milestone Om2008.8 deleted

comment:9 Changed 11 years ago by wjbaird

I've encountered the same problem - using a mostly stock OM2008.8 install on a freerunner. a sample PDU is

0791446742949940040BD0C7F7FBCC2E0300008080919075106935D2723BED2697E53A10BD3CA787400010B55E0605EB67502C078AC1743059B80D6A8162311D4C166E8350D7B77C9D02

what I see is very similar to the already attached screenshot.

comment:10 Changed 11 years ago by tick

  • Cc tick@… added

comment:11 Changed 11 years ago by tick

  • Owner changed from zecke to tick

comment:12 Changed 11 years ago by tick

Hi Odlg,

Sorry I cannot setup google calendar SMS service in Taiwan now. (Google not support yet. (Though it say yes, but I cannot find yet.))

Is your content of SMS also wrong (I cannot read Latin), or just sender field wrong?

For sender part you can find from [3Gpp TS 23.040 9.1.2.3]
The SMS from google show 040DD0 <- D == 1101 -> 101 is the type of message. It's Alphanumber, (Coded according to [3GPP TS 23.038 6.2.1] GSM 7-bit default alphabet)
Alphanumber, Unknow plan

For the SMS avanc provide which works is 040C91 in which shows:
International number, ISDN/telephone numbering plan.

Yes, it is not implemented in qmodemsmsreader.cpp , I'll dealing with this.

Cheers,
Tick

comment:13 Changed 11 years ago by odlg

Yes, the content of the sms is wrong, and the sender field has some garbage in it.

The PDU:

07918167830071F1040BD0C7F7FBCC2E030000808010800120804AD0473BED2697D9F3B20E644CCBDBE136835C6681CCF2B20B147381C2F5B30B04C3E96630500B1483E96030501A34CDB7C5E9B71B847AB2CB2062987D0E87E5E414

should provide this sms content:

Påmindelse: Firmamøde, fre. 1. aug. 08:30 - 10:00 i Symbion (Ole Dalgaard)

comment:14 follow-up: ↓ 15 Changed 11 years ago by tick

some updates

I think qsmsmessage.cpp:address(bool) 1854 has do this.
But seems something wrong with that.
I guess QByteArray bytes = expand7Bit( getOctets( len ) ); shall not getOctets.
deep digging

comment:15 in reply to: ↑ 14 Changed 11 years ago by tick

Replying to tick:

some updates

I think qsmsmessage.cpp:address(bool) 1854 has do this.
But seems something wrong with that.
I guess QByteArray bytes = expand7Bit( getOctets( len ) ); shall not getOctets.
deep digging

ah... yes it should. :P

comment:16 follow-up: ↓ 17 Changed 11 years ago by tick

I think in Qtopia it will to view the semi-octet address length as the string length of the 7-packed string.
(The unit of getOctets is byte ~_~)
I think its wrong.
according to 3GPP-TS 23.40 9.1.2.5
The Address-Length field is an integer representation of the number of useful semi-octets within the Address-Value
field, i.e. excludes any semi octet containing only fill bits.

for 'sipgate' shall be 14 octets with 8 bit pack and 13 with 7-pack
that why the address length is 0x0D.

In Qtopia it tried to access 13 bytes instead, I think that's why I can see 'google' on the screenshot and some other stuff, and why messages are ruined.

The following patch is my fix through my guess, it is *NOT* tested yet.
Holger would you test this for me?
I cannot reproduce this issues easily in Taiwan, and only can guess.
However, if the length of message is odd, it seems will cause trouble. (Still need to check)

diff --git a/src/libraries/qtopiaphone/qsmsmessage.cpp b/src/libraries/qtopiaphone/qsmsmessage.cpp
index 1b05131..4adbef3 100644
--- a/src/libraries/qtopiaphone/qsmsmessage.cpp
+++ b/src/libraries/qtopiaphone/qsmsmessage.cpp
@@ -1872,9 +1872,7 @@ QString QPDUMessage::address(bool SCAddress)

skipOctet();
if ( !SCAddress ) {

  • if ( at != SMS_Address_AlphaNumeric ) {
  • len = len / 2 + (len % 2);
  • }

+ len = len / 2 + (len % 2);

} else {

len--;

}

comment:17 in reply to: ↑ 16 Changed 11 years ago by tick

However, if the length of message is odd, it seems will cause trouble. (Still need to check)

Hmmm... It seems okay. I should read even octets for one 7Bit-packed data. So if the data says 13, actually with 7Bit pack it will padding to 14 octets ie. 7 bytes.
So I guess the patch may be okay.

comment:18 Changed 11 years ago by tick

I think it works.
--

tick@tock:~/work/qtopia-build/src/applications/smsparser>./smsparser 0791447758100650040DD0F334FC1CA6970100008080312170224008D4F29CDE0EA7D9
ByteArray? 0791447758100650040DD0F334FC1CA6970100008080312170224008D4F29CDE0EA7D9
Sender: "sipgate@"
Text: "Testmail"
tick@tock:~/work/qtopia-build/src/applications/smsparser>./smsparser 0791447758100650040C9194714373238200008080312160304019D4F29C0E6A97E7F3F0B90CB2A7C3A0791A7E0ED3CB2E
ByteArray? 0791447758100650040C9194714373238200008080312160304019D4F29C0E6A97E7F3F0B90CB2A7C3A0791A7E0ED3CB2E
Sender: "+491734373228"
Text: "Test message via sipgate."
tick@tock:~/work/qtopia-build/src/applications/smsparser>./smsparser 07918167830071F1040BD0C7F7FBCC2E0300008080203201208078D0473BED2697D9F3B20E442DCFE9A076793E0F9FCBA07B9A8E0691C3EEF41C0D1AA3C3F2F0985E96CF75A00EE301E22C1C2C109B217781642E50B87E76816433DD0C066A81E60CB70B347381C2F5B30B
ByteArray? 07918167830071F1040BD0C7F7FBCC2E0300008080203201208078D0473BED2697D9F3B20E442DCFE9A076793E0F9FCBA07B9A8E0691C3EEF41C0D1AA3C3F2F0985E96CF75A00EE301E22C1C2C109B217781642E50B87E76816433DD0C066A81E60CB70B347381C2F5B30B
Sender: "Google"
Text: "Pmindelse: Test message with danish characters: , lr. 2. aug. 23:30 - sn. 3. aug."

comment:19 Changed 11 years ago by tick

  • Status changed from accepted to in_testing

commit as dbd50b81c36ef87e26ad230b50dec256adbb7cb1

comment:20 Changed 11 years ago by tick

  • Status changed from in_testing to assigned

It's not in image yet, wait it in image then switch to in_testing

comment:21 Changed 11 years ago by zecke

  • Status changed from assigned to in_testing

Okay, this should be in tomorrow's build (wherever that shows up).

comment:22 Changed 11 years ago by odlg

My Freerunner is up2date from http://people.openmoko.org/~zecke/om2008.8-dev/ as of now, got a lot of qtopia updates. It does not work yet. See the screenshots.

Changed 11 years ago by odlg

Changed 11 years ago by odlg

comment:23 Changed 11 years ago by tick

It was built at Aug 23, not Aug 27

comment:24 Changed 11 years ago by zecke

Sorry, I'm only on a GPRS line and managed to upload the packages to om2008.8-testing. Could you please test that? You should get 4.3.2+git413+13101ab6ddaec380871ae8021a92140af526c4df-r39.1 as version for Qtopia.

comment:25 Changed 11 years ago by odlg

OK, that seems to help, new SMS's are shown correct, old SMS's seems to be a mix of correctly shown SMS's and non-correct.

comment:26 Changed 11 years ago by wjbaird

just did an opkg update from the testing tree, and sms's from google are working fine for me.

Thanks!!!

comment:27 Changed 11 years ago by zecke

You might want to delete the current "inbox" (~/root/Applications/qtmail IIRC) and this way force qtmail to "reparse it".

comment:28 Changed 11 years ago by zecke

  • Status changed from in_testing to closed
  • Resolution set to fixed

With this http://git.openmoko.org/?p=qtopia.git;a=commit;h=980968fa7e7362b5450c6d7b147e287efe3a20d8 we have regression testing for SMS parsing in place now. Let us see what the change broke and then fix it.

Thanks tick.

Changed 10 years ago by Treviño

Badly decoded SMS

comment:29 Changed 10 years ago by Treviño

I reopen this ticket, since as you can see from this attached screenshot I got this badly decoded SMS:
https://docs.openmoko.org/trac/raw-attachment/ticket/1728/Screenshot-24.png
I don't really know what's inside... A PDU I can get from logread is (not coming from the SMS above, btw):

Sep 10 00:15:42 om-gta02 user.notice root: AtChat :  F : "0791934329002000840C9193335522291600008090608155708099D6A2B32825264FA098EC95033DA545902C064A3A41D3A236F9741641D264D5997C3A8B206A71E84C0E83A0A750C84C1E83D4A7341964399F4ED0F4E97C8282CD66713A2D8282D369D1A92DBA40C46733

PS: Why not storing also the PDU not to loose the message for decoding errors?

comment:30 Changed 10 years ago by tick

Is the following decode right? (I don't understand it at all :P )
If is correct, I think you encounter another issue, or your qtopia version is not the same as ours.

./smsparser 0791934329002000840C9193335522291600008090608155708099D6A2B32825264FA098EC95033DA545902C064A3A41D3A236F9741641D264D5997C3A8B206A71E84C0E83A0A750C84C1E83D4A7341964399F4ED0F4E97C8282CD66713A2D8282D369D1A92DBA40C46733

ByteArray? 0791934329002000840C9193335522291600008090608155708099D6A2B32825264FA098EC95033DA545902C064A3A41D3A236F9741641D264D5997C3A8B206A71E84C0E83A0A750C84C1E83D4A7341964399F4ED0F4E97C8282CD66713A2D8282D369D1A92DBA40C46733
Sender: "+393355229261"
Text: "VENERDI' 12/9 ORE 21 IN SEZIONE RIUNIONE TECNICA OBBLIGATORIA,NON SONO AMMESSE ASSENZE. DOM"

comment:31 Changed 10 years ago by Treviño

Yes it is... But that PDU is not the PDU of the badly-decoded SMS I've screen-shotted above. Unfortunately I can't find that PDU!

Any way to recover it? Thanks!

comment:32 Changed 10 years ago by wjbaird

Yup - I just started getting garbled messages from google again last night. It doesn't seem like all of them are garbled - I last upgraded (via OM2008.8 testing) yesterday, but I got some google SMSs yesterday and today that were fine, but the last two that I got today were garbled.

I was only able to find the PDU for one of them: 0791446742949940040BD0C7F7FBCC2E0300008090013102126964D2723BED2697E53A50D10AA296C36D50B35CA6A7DD671000742D9341D3321C14830CDE06B54032DD0C066F83D26ED0531E1EB3CBA066B94C4FBBCF7076785C06DD6A379B0D067A816EB59BCD0603A1AE

However, when I tried to decode this at http://twit88.com/home/utility/sms-pdu-encode-decode I see the same thing I see on the OM - the first half of the message is fine, and the last half garbled:

Reminder: EV Team Meeting @ Wed Sep 1Δ¥oùj$ΦS&#8364;¥¥oùJvùzΛåßf.ùj,.&Ov>ùgÅß.ù:+;33¥¥z$:+;33¥¥B:é

I assume this means that the SMS itself was garbled by google? Or could something at a lower level be scrambling the PDU itself?

Note: See TracTickets for help on using tickets.