EN
FR
DE
ES
NO
SV
FI
DA
LV
The biggest challenge when trying to implement automation for game movements in a real gameplay environment is the dynamic nature of the game, which makes it difficult to make predictions. Random in-game events can happen at unexpected times, which might interfere with the testing process. That is the reason why we need to introduce some sort of way to analyze what is happening on the screen and respond according to the event.
In this case, using AI will make things a lot easier because we can add custom movement logic that will only be executed if that event happens. For instance, if there is an obstacle in the way of our playable character, we can tell it to either jump over the obstacle or move around it. It still involves sequentially defining an action to be taken, but this approach makes testing a bit more organic, meaning, we can handle some unpredictable events. Using AI in this way to recognize what is happening on the screen makes it a lot easier to add automated in-game movements, as we can handle unpredictable cases and act upon them.
That said, it is important to note that even with AI, some events might be difficult to handle depending on the in-game situation. For example, fast-paced games or poorly lit areas can make object detection difficult for the AI. Even for slower-paced games, it is important to turn off some post-processing effects, such as motion blur, which otherwise might interfere with the recognition process.
Understanding game mechanics
Movement is a fundamental aspect of video games, allowing players to navigate and interact with the game world. It is a crucial element of gameplay, as it enables players to explore, combat, and solve puzzles. Movement mechanics can range from simple actions like walking and running to complex actions like jumping, climbing, and flying. The smoothness and responsiveness of movement are essential for creating an immersive and enjoyable gaming experience.
In many games, movement is not just about getting from point A to point B; it’s about how players engage with the environment and the challenges they face. For instance, in platformers, precise jumping and timing are critical, while in open-world games, the ability to traverse vast landscapes seamlessly enhances the sense of exploration. Combat-focused games often require quick reflexes and strategic movement to outmaneuver opponents. Thus, the design and implementation of movement mechanics can significantly impact the overall gameplay experience.
How can in-game movements be automated?
In order to automate in-game movements, we first need to understand the sequence of actions that need to be taken algorithmically. To make it easier to understand the logic, we can draw a flowchart to visualize this algorithm.
Afterward, we need to choose which programming language to use. A very easy choice is Python because it offers a sizable list of libraries that makes achieving our goal much easier. Do note that while these libraries let us sequentially automate some movement, there is no integration with AI. We need to create the integration ourselves. Feedback from gamers regarding movement issues should be prioritized in development to ensure a better gaming experience.
Challenges with Python libraries for motion capture
For automating keyboard actions, we can utilize the Python library PyAutoGui. Even though it works flawlessly with simple use cases, like entering text into a browser or a simple windowed application, it has issues working with in-game character controls. This is related to how keyboard events are processed when a game window is in focus. To circumvent this issue we can use another library called PyDirectInput, which uses the Windows native API to simulate an actual keyboard event. That addresses the keyboard event registration issue, but do note that this is a Windows-specific solution and alternatives must be used if GNU/Linux or MacOS is your target operating system. In some cases, integrating motion capture technology can enhance the accuracy of automated movements.
Similarly, for mouse actions, PyAutoGUI does work for simple use cases and even allows us to interact with the game UI, but unfortunately, it cannot control playable characters, hence these events are simply ignored due to how they are processed. At the time of writing this article, PyDirectInput does not solve our issue, and mouse move actions are simply ignored the same way they are when using PyAutoGUI. There is a workaround that works for a specific set of games that give us a “Raw input” mouse setting, but that limits our ability to test a broad range of games. Unfortunately, there is no easy-to-use library that can let us control mouse actions in a way that PyAutoGUI does, and as such, we are limited to using the native Windows API. To access the Win32 API, we can use a library called PyWin32.
Utilizing the Windows Native APIs with Python
While using native APIs directly, most of the movement logic has to be implemented by us to ensure a seamless playing experience. Overall, implementation is relatively straightforward, but some tricky details can make things complicated. For example, if mouse smoothing is necessary, we also need to implement an algorithm that is going to do that for us. Otherwise, mouse movements will be instant since native APIs do not provide a smoothing feature. In this particular case, mouse click and scroll commands will be directly forwarded to the Win32 API, and, for mouse move commands, we will implement some smoothing so that the mouse movements are not instantaneous.
The final step is to delegate which library is going to be responsible for which commands. For the most part, PyDirectInput and PyWin32 are going to do the bulk of the work by executing keyboard and mouse commands respectively, but PyAutoGUI will act as a fallback for other operating systems, and cases where some commands could not be implemented using PyDirectInput or PyWin32.
Video games as contained universes
Video games are self-contained universes with their own rules, physics, and mechanics. Each game has its unique laws of physics and mechanics that govern its world, allowing players to explore and interact with the environment in a specific way. The game world is a virtual space that is designed to be navigated and explored, with its own geography, architecture, and obstacles.
These virtual worlds are crafted to immerse players in a believable and engaging environment. The physics engine of a game dictates how objects move and interact, creating a sense of realism or fantasy depending on the game’s design. For example, a game set in a medieval fantasy world might have different gravity and collision rules compared to a sci-fi game set in space. Understanding these unique mechanics is essential for both players and developers to fully appreciate and utilize the game world.
Navigating the game world
Navigating the game world is a critical aspect of gameplay, as it allows players to explore, discover new locations, and progress through the game. The game world is often designed to be intuitive and easy to navigate, with clear paths, landmarks, and visual cues to guide the player. However, some games may feature more complex navigation systems, such as puzzles, mazes, or open-world environments, which require players to use their problem-solving skills and creativity to navigate.
In open-world games, players are often given the freedom to explore vast landscapes at their own pace, uncovering secrets and completing quests. These games rely on a well-designed map and navigation system to help players find their way. On the other hand, puzzle-based games might challenge players with intricate mazes or hidden paths that require careful observation and strategic thinking. Regardless of the complexity, effective navigation design enhances the player’s ability to engage with the game world and enjoy the gameplay experience.
By understanding the mechanics and navigation of game worlds, developers can create more immersive and enjoyable experiences for players, while players can better appreciate the intricacies of the virtual environments they explore.
Video games as contained universes
Video games are self-contained universes with their own rules, physics, and mechanics. Each game has its unique laws of physics and mechanics that govern its world, allowing players to explore and interact with the environment in a specific way. The game world is a virtual space that is designed to be navigated and explored, with its own geography, architecture, and obstacles.
These virtual worlds are crafted to immerse players in a believable and engaging environment. The physics engine of a game dictates how objects move and interact, creating a sense of realism or fantasy depending on the game’s design. For example, a game set in a medieval fantasy world might have different gravity and collision rules compared to a sci-fi game set in space. Understanding these unique mechanics is essential for both players and developers to fully appreciate and utilize the game world.
Navigating the game world
Navigating the game world is a critical aspect of gameplay, as it allows players to explore, discover new locations, and progress through the game. The game world is often designed to be intuitive and easy to navigate, with clear paths, landmarks, and visual cues to guide the player. However, some games may feature more complex navigation systems, such as puzzles, mazes, or open-world environments, which require players to use their problem-solving skills and creativity to navigate.
In open-world games, players are often given the freedom to explore vast landscapes at their own pace, uncovering secrets and completing quests. These games rely on a well-designed map and navigation system to help players find their way. On the other hand, puzzle-based games might challenge players with intricate mazes or hidden paths that require careful observation and strategic thinking. Regardless of the complexity, effective navigation design enhances the player’s ability to engage with the game world and enjoy the gameplay experience.
By understanding the mechanics and navigation of game worlds, developers can create more immersive and enjoyable experiences for players, while players can better appreciate the intricacies of the virtual environments they explore.
What are some caveats of automated game world testing?
As much as automated game testing using AI sounds exciting, there are some caveats that need to be taken into account before integrating it into your workflow. The main problem stems from inconsistencies when detecting objects, which may lead to a test ending abruptly if it fails to detect a certain object that is expected to be in the sequence. We can make it handle a decent amount of random in-game events, but there is still a chance of something happening that will block the sequence from progressing.
Also worth noting is that lighting, screen clarity, and game pace all play a huge role in determining whether or not AI is a viable option. If the game takes place in a very dimly lit environment, has any post-processing effects enabled, such as motion blur, chromatic aberration, and lens flares, or even in the case that the game viewport moves very quickly, the object recognition process could falter significantly.
While a single model may not work for all games, there are some implementations, such as YOLOv7, that work relatively well for certain cases. However, some games require a purposefully trained model in order for it to work on very unique objects.
Implementing in-game movements ourselves is a tricky endeavor as there are a lot of caveats and details that need to be taken into account. At the time of writing this article, there is no universal all-in-one solution to control all possible movement types on all operating systems, so it is necessary to combine multiple libraries. Although as game testing in this manner is still in its very early phase, there are bound to be challenges. If we see wider industry adoption of similar testing methodologies, we can expect to see more sophisticated tools becoming available. Despite these challenges, the possibilities for enhancing game testing through automation are vast and continually evolving.
Final thoughts
Automating in-game movements presents unique challenges due to the unpredictable nature of gameplay, but integrating AI and specialized Python libraries can help streamline the process. By leveraging tools like PyAutoGUI, PyDirectInput, and PyWin32, testers can create automated workflows that handle many in-game scenarios. However, limitations such as object detection inconsistencies, lighting conditions, and game-specific variables must be carefully considered.
As game testing evolves, we can expect more advanced AI-driven solutions to enhance automation and improve accuracy. While there is no one-size-fits-all approach today, combining AI with robust scripting techniques is a step toward more efficient and scalable game testing. For those looking to optimize their automation strategies, staying updated with emerging technologies and best practices will be key to success.
Check out our artificial intelligence testing services page for more information on how we can help you improve your AI product, or contact us for more details.
