{"id":9034,"date":"2025-05-30T07:40:51","date_gmt":"2025-05-30T02:10:51","guid":{"rendered":"https:\/\/pheonixsolutions.com\/blog\/?p=9034"},"modified":"2025-05-30T07:40:58","modified_gmt":"2025-05-30T02:10:58","slug":"ways-to-overcome-the-obstacles-associated-with-sms-messaging-load-testing-at-10-million-sends","status":"publish","type":"post","link":"https:\/\/pheonixsolutions.com\/blog\/ways-to-overcome-the-obstacles-associated-with-sms-messaging-load-testing-at-10-million-sends\/","title":{"rendered":"Ways To Overcome The Obstacles Associated With SMS Messaging Load Testing At 10 Million Sends"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><strong>Overview\u00a0\u00a0<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Communication as an operation comes hand-in-hand with load and stress testing. Not long ago, I had the honor of executing an SMS use case of sending 10 million messages in SMS: performing sending it in mass. Obviously, every test was beyond the bare minimum. Dominant successes needed to be achieved on benchmarks like performance, scalability, and resilience. In this blog post, I will outline the obstacles faced during the high throughput SMS tests and explain the ways these challenges were met on a multidisciplinary planning, and tooling basis with coordination and foundational frameworks.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Challenge #1 Infrastructure bottlenecks<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Challenge:\u00a0<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Messaging 10M Concurrent SMS Messages comes with great database requisites, throughput limits, computer requisites, as well as bandwidth. Certain blocks\/environments like our SMS messaging test environment were \u2013 and to some degree still are: Not set for this framework, causing regular timeouts and failures in services.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Result:\u00a0<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Increased the scope of monitoring the instance as a result of working together with Devops to enforce Auto-scaling groups and Load balanced instances.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Challenge #2: Queue &amp; Throughput Limitations<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Problem:\u00a0<\/strong>\u00a0<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u00a0The messaging system depended on message queues (like RabbitMQ). Each of these queues reached their individual throughput ceilings which created performance lags in message processing upto 4:30  hours for 10 million SMS at the time interval of 10 to 30 minutes, and it reaches the CPU Usage of 100%.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Solution:\u00a0\u00a0<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>&nbsp;Implemented batch message publishing for more efficient queue operations.&nbsp;&nbsp;<\/li>\n\n\n\n<li>&nbsp;Prefetch limits, acknowledgment methods, and consumer scaling were optimized within the queue configurations.<\/li>\n\n\n\n<li>Choke points were actively identified using Prometheus + Grafana alongside real-time monitoring to expedite the flow of systems under strain.&nbsp;&nbsp;<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Challenge #3: Outbox Timeout at Scale<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Problem:\u00a0\u00a0<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;During the test, once I scaled up to 10 million contacts, the outbox service became increasingly unresponsive and started returning timeout errors. Messages encountered a bottleneck in the system and were unable to be transitioned to the next processing stage.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Solution:\u00a0\u00a0<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Developments of this swirling problem were attributed on-the-spot to the message dispatching layer alongside DevOps which designated their control sliders for the outbox servicing system.<\/li>\n\n\n\n<li>Set adjusted timeouts on other dependent services while also increasing overall box service throughput.&nbsp;&nbsp;<\/li>\n\n\n\n<li>The system metrics and logs were actively monitored post-tuning to ensure the desired velocity of flow was attained and thus validate the outcome.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-29.png\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"575\" src=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-29-1024x575.png\" alt=\"\" class=\"wp-image-9036\" srcset=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-29-1024x575.png 1024w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-29-300x168.png 300w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-29-768x431.png 768w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-29-1536x862.png 1536w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-29-534x300.png 534w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-29.png 1644w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Challenge #4: Test Data Management\u00a0<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Problem:<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;Duplicates were inefficiently managing and contending with the test data for 10 million unique SMS messages. Overlapping numbers or malformed payloads could skew results.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Solution:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Used Python scripts to automate the creation of unique, valid phone numbers alongside distinct message templates.&nbsp;<\/li>\n\n\n\n<li>Before dispatch, monitored payload integrity with purpose built data validators.\u00a0<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Challenge #5: Performance Metrics Monitoring\u00a0<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Problem:<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">&nbsp;Collecting meaningful parameters and metrics (latency, delivery rate, failure rate) with such a massive volume was disorderly at first.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Solution:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>&nbsp;Applied APM integration for backend and message path tracing with New Relic, DataDog and other similar tools.&nbsp;<\/li>\n\n\n\n<li>Set the following key SLAs: end-to-end latency, SMS delivery success rate, and retry rate.&nbsp;<\/li>\n\n\n\n<li>With custom dashboards, filtered non-relevant metrics and visualized them in near real time.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-31.png\"><img decoding=\"async\" width=\"1024\" height=\"286\" src=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-31-1024x286.png\" alt=\"\" class=\"wp-image-9038\" srcset=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-31-1024x286.png 1024w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-31-300x84.png 300w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-31-768x214.png 768w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-31-850x237.png 850w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-31.png 1118w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Challenge #6: Failure Recovery and Retry Mechanisms<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Problem:\u00a0<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The primary reason was a great deal of temporary network malfunctions and external SMS gateway problems.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Solution<\/strong>:<\/p>\n\n\n\n<div class=\"wp-block-group is-vertical is-layout-flex wp-container-core-group-is-layout-2c90304e wp-block-group-is-layout-flex\">\n<ul class=\"wp-block-list\">\n<li>Coordinated with developers on creating simulated chaos testing failures on the gateway.<\/li>\n\n\n\n<li>Validated attempts at cancellation logic verification and exponential risk inflation gradual increase algorithms.<\/li>\n\n\n\n<li>Confirmed configuration completion on message queues that cannot be recovered sent to the dead letter queue.<\/li>\n<\/ul>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Challenge #7: Cross-Team Collaboration<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Problem:<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The scope of such a test was large scale, capturing the synergy between multiple participants at QA, DevOps, Backend.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Solution:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Developed war rooms and live dashboards for the duration of the testing window.<\/li>\n\n\n\n<li>Designed ACTs for resolution tracking, escalation, and rapid issue resolution.<\/li>\n\n\n\n<li>Compiled logs of timestamped root causes for all issues captured for post-mortem review.<\/li>\n\n\n\n<li>Verified fundamental initial pre-test strategy derived observations.&nbsp;<\/li>\n\n\n\n<li>Planning determines everything \u2013 environment readiness, data strategy, and coordination need to be locked before the test.<\/li>\n<\/ul>\n\n\n\n<div class=\"wp-block-group is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"wp-block-paragraph\"><strong>Challenge #8 : Test Plan<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Problem:<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In order to find out the capacity of our system, I need to test sending 10 M messages and monitor the cpu usage after the time intervals of 10 and 20 minutes. This is my test strategy.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-37.png\"><img decoding=\"async\" width=\"857\" height=\"346\" src=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-37.png\" alt=\"\" class=\"wp-image-9044\" srcset=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-37.png 857w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-37-300x121.png 300w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-37-768x310.png 768w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-37-743x300.png 743w\" sizes=\"(max-width: 857px) 100vw, 857px\" \/><\/a><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-35.png\"><img loading=\"lazy\" decoding=\"async\" width=\"893\" height=\"351\" src=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-35.png\" alt=\"\" class=\"wp-image-9042\" srcset=\"https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-35.png 893w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-35-300x118.png 300w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-35-768x302.png 768w, https:\/\/pheonixsolutions.com\/blog\/wp-content\/uploads\/2025\/05\/image-35-763x300.png 763w\" sizes=\"(max-width: 893px) 100vw, 893px\" \/><\/a><\/figure>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Conclusion\u00a0<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Quality Assurance (QA) has so far remained a one shot deal with me for testing 10 million SMS messages, honing and blending all the skills together in single unit tests. It never failed to challenge me; rather, it was deeply gratifying. This taught us a myriad of lessons about system scalability, fault tolerance, and teamwork. In my experience, these multifaceted challenges remind me how critical QA is in a business and its sophisticated technologies. It also reminds us about the importance of performance surpassing targets regarding said objectives. As gatekeepers of robust systems, we uncover shortcomings through agile drills.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Overview\u00a0\u00a0 Communication as an operation comes hand-in-hand with load and stress testing. Not long ago, I had the honor of [&hellip;]<\/p>\n","protected":false},"author":506,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1022],"tags":[],"class_list":["post-9034","post","type-post","status-publish","format-standard","hentry","category-web-architecture","psol-cat-web-architecture"],"jetpack_publicize_connections":[],"jetpack_shortlink":"https:\/\/wp.me\/phn2x7-2lI","jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/posts\/9034","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/users\/506"}],"replies":[{"embeddable":true,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/comments?post=9034"}],"version-history":[{"count":0,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/posts\/9034\/revisions"}],"wp:attachment":[{"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/media?parent=9034"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/categories?post=9034"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pheonixsolutions.com\/blog\/wp-json\/wp\/v2\/tags?post=9034"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}