Skip to content

tom_modem: gate Turkish GSM-7 shift table behind UDH language IE - #259

Open
ozgunokan wants to merge 3 commits into
FUjr:mainfrom
ozgunokan:sms-tr-charset-fix-v2
Open

tom_modem: gate Turkish GSM-7 shift table behind UDH language IE#259
ozgunokan wants to merge 3 commits into
FUjr:mainfrom
ozgunokan:sms-tr-charset-fix-v2

Conversation

@ozgunokan

@ozgunokan ozgunokan commented Aug 30, 2026

Copy link
Copy Markdown

This replaces the Turkish SMS support from PR #256/#257 (closed as
duplicate) with a version that addresses the review feedback:

  • Turkish extension mappings moved to a separate table, only selected
    when the PDU's UDH explicitly signals Turkish via the National
    Language Locking/Single Shift IE (0x24/0x25, language id 0x01).
    Non-Turkish GSM 7-bit messages now decode exactly as before.
  • Added a bounds check in sms_send() for the previously-unvalidated
    PDU length going into the fixed 512-byte pdu_hex buffer.

Kernel 6.18 changes are intentionally left out of this PR per your
request to split unrelated changes -- this is SMS-only.

Tested on Arcadyan AW1000 (Quectel RG500Q-EA), IPQ807x.

@FUjr

FUjr commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Thank you for revising the Turkish SMS support. I found two remaining blockers before this PR can be merged:

  1. The PR is still not SMS-only. Although the description says the kernel 6.18 changes were removed, the current PR diff still adds 100-kernel-6.18-compat.patch and 200-nss-registration-retry.patch (153 lines in total), inherited from commits 601afcd, 6d57ef5, and 603d895. Please recreate or rebase the branch on the latest main so that the PR contains only the intended SMS changes. The included compatibility patch also stores per-device bottom-half state in the global usbnet_bh_qmapdev and usbnet_orig_bh_func variables, which is unsafe when more than one device is present.

  2. The Turkish locking-shift and single-shift mechanisms are still conflated. udh_has_turkish_shift() returns the same boolean for IEI 0x24 and 0x25, while G7bitToAscii() only switches the extension table used after ESC. A Turkish locking-shift IE must select the Turkish locking/base alphabet for normal septets, whereas a single-shift IE selects the table used after ESC; both IEs may also appear in the same message. For example, with a Turkish locking-shift IE, septet 0x0B is currently decoded through the default alphabet as Ø instead of Turkish Ğ.

Please track the locking and single-shift selections separately, apply them to the base and extension lookups respectively, and add focused tests for locking-only, single-only, and combined UDH cases. Thanks again for the contribution and the device testing.

ozgunokan and others added 3 commits September 8, 2026 10:22
@ozgunokan @FUjr
tom_modem: add Turkish character support for SMS
82fff5d
@ozgunokan @FUjr
tom_modem: gate Turkish GSM-7 shift table behind UDH language IE
49e2f81 
Addresses review feedback on FUjr#257/FUjr#256: the Turkish national language
extension mappings were previously written directly into the shared
gsm7bits_extend_to_latin1 table, so every GSM 7-bit message decoded
through it regardless of language -- reserved extension codes used by
other languages/future 3GPP additions could have been misdecoded as
Turkish characters.

This moves the Turkish mappings into a separate turkish_extend_to_latin1
table and adds udh_has_turkish_shift(), which walks the PDU's User Data
Header IEs and only selects that table when a National Language Locking
Shift (IEI 0x24) or Single Shift (IEI 0x25) IE explicitly selects Turkish
(language id 0x01, per 3GPP TS 23.038 table 6.2.1.2.5). Every other
message continues to decode through the original, unmodified table.

Also adds a bounds check in sms_send() (operations.c): profile->sms_pdu
was hex-encoded into a fixed 512-byte pdu_hex buffer with no upper
bound on the source PDU length, which could overflow the stack buffer
for a long/malformed PDU. This rejects PDUs longer than 255 chars
(the max that fits pdu_hex[512] as hex + terminator) up front.

Tested on Arcadyan AW1000 (Quectel RG500Q-EA), IPQ807x: Turkish SMS
with the language shift IE set decode correctly; a batch of non-Turkish
GSM 7-bit messages (EN/DE test SMS) decode unchanged before and after
this patch.
@FUjr
tom_modem: fix Turkish UDH parsing and PDU validation
0a5b566
@FUjr
FUjr force-pushed the sms-tr-charset-fix-v2 branch from c8da16a to 0a5b566 Compare September 8, 2026 10:51
@FUjr

FUjr commented Sep 8, 2026

Copy link
Copy Markdown
Owner

I have now rebuilt this branch on the latest main and kept it strictly SMS-only.

The changes include:

  • separate Turkish National Language Locking Shift (0x24) and Single Shift (0x25) handling;
  • complete UDH IE walking, so concatenation metadata is found correctly even when language/application IEs appear before or after it;
  • preservation of the standard GSM-7 tables unless the corresponding Turkish UDH IE is present;
  • validation of PDU length and hexadecimal input before decoding into the fixed buffer;
  • focused tests covering locking-only, single-only, combined, reordered-IE, and malformed-UDH cases.

The updated head is 0a5b566. The NSS/kernel changes have been removed from this PR and are being handled separately in PR #256.

Would you be comfortable merging this SMS-only version in its current form?

@FUjr FUjr mentioned this pull request Sep 8, 2026
Open
@ozgunokan

ozgunokan commented Sep 8, 2026

Copy link
Copy Markdown
Author

Thanks for the quick turnaround. Before giving a final go-ahead, I want to double-check the locking vs. single-shift decoding against a couple of manual test PDUs and confirm the diff is fully clean of the kernel/NSS changes. I'll follow up within the next day with a confirmation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ozgunokan @FUjr