With the proliferation and the critical role of Internet of Things (IoT) in various domains, rigorous testing techniques are required for improving their quality. Several review studies have examined IoT topology and analyzed different layers of the IoT architecture. These assessments include synthesis on the application layer protocols, but do not specifically study the application layer protocol standalone. As a result, this work focuses on unique application-layer protocol-specific testing approaches, using different tools and frameworks, and systematically analyzes the challenges in an in-depth manner. There is a need for a specific and deeper analysis of the application layer IoT protocol testing state of the art. This paper aims to bridge this gap by conducting a Systematic Literature Review (SLR) to provide a deeper and concise picture of the state of the art in testing IoT application layer protocols. We consider studies published between 2010 and 2025, resulting in 47 primary studies selected from 822 retrieved papers after systematic screening. These studies have been retrieved from eight major digital libraries. Following the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines, we discuss and synthesize our findings on the different types of testing, explore the most promising approaches, frameworks and tools, trends, highlight the main challenges, and identify open research issues and directions in testing IoT application layer protocols.