Testing gameplay I

Tests are crucial part of any system; tests help us make sure the code we ship works as intended. Ah yes… and yet, there are bugs left. And a lot of the times, it’s the players that discover the bugs: strange bugs; complicated and unexpected sequences of actions, leading to unanticipated, unexplored state, leading to bugs. In KOPI, we wanted to make sure that we catch as much as possible before releasing the game to our beta players, never mind the wider audience.

Structure

Our KOPI game code is made up of three modules–Unreal Engine modules.

• KopiEditor
• KopiGame
• KopiTests

The KopiGame contains the core of our game, it is the module that ends up in the shipping version of the game. The KopiEditor module contains developer quality-of-life tools (think validators, convenient views, test launchers, etc.); finally, KopiTests is where the testing code lives. When we build non-Shipping build, KopiTests and KopiEditor are excluded.

The motivation behind this structure is to ensure that we can keep the main codebase clean and ready to be built into a shipping version at any stage; but also to ensure it can be tested during development. The tests are black-box tests that exercise the KopiGame functionality without the KopiGame knowing it is being tested.

Assertions

A part of the tests are assertions and state-check code that ensures that individual elements are in well-known state. We simply use conditional compilation with the #if WITH_TESTS ... #endif blocks. The WITH_TESTS is not defined in the shipping build, making our shipping build as streamlined as possible. However, unit-level assertions and state validation is nowhere near enough; it is necessary to exercise the game’s code to ensure that we explore the entire (or as much as possible) of the game state space.

KopiTests

The challenge in implementing our black-box tests was to make sure that we can inject the necessary test structure without having to modify the KopiGame module code or its content. The current solution uses a UWorldSubsystem that sets the test controllers on the AKopiBaristas–the characters that a human player would control.

UCLASS()
class KOPITESTS_API UKopiTestWorldSubsystem : public UWorldSubsystem {
	GENERATED_BODY()
public:
  virtual void OnWorldBeginPlay(UWorld &InWorld) override;
};

The implementation of the OnWorldBeginPlay method injects the test controllers, and begins “standard” gameplay.

void UKopiTestWorldSubsystem::OnWorldBeginPlay(UWorld &InWorld) {
	Super::OnWorldBeginPlay(InWorld);

	if (AKopiGameModeGame *GM = InWorld.GetAuthGameMode<AKopiGameModeGame>();
	    GM != nullptr) {
		GM->Test_OnMatchEnded.BindUObject(this, &ThisClass::OnMatchEnded);
	}

	TArray<FString> Tokens;
	TArray<FString> Switches;
	TMap<FString, FString> Params;
	UKismetSystemLibrary::ParseCommandLine(FCommandLine::Get(), 
	  Tokens, Switches, Params);
	bAutoPlayNext = Switches.Contains("TestAutoPlayNext");
	if (const FString *Drivers = Params.Find("TestDrivers"); Drivers != nullptr) {
		TArray<FString> DriverIds;
		Drivers->ParseIntoArray(DriverIds, TEXT(","), true);
		for (const FString &DriverId : DriverIds) {
			PlayerDriverIds.AddUnique(FCString::Atoi(*DriverId));
		}
	}
	if (const FString *ChaosParameter = Params.Find("TestChaos"); 
	  ChaosParameter != nullptr) {
		Chaos = FCString::Atof(**ChaosParameter);
	} else {
		Chaos = 0.f;
	}

	AGameStateBase *GameStateBase = InWorld.GetGameState();
	if (GameStateBase == nullptr) {
		return;
	}
	if (!GameStateBase->IsA<AKopiGameStateGame>()) {
		return;
	}

	FString MapName = InWorld.GetMapName();
	if (MapName.StartsWith(TEXT("UEDPIE_0_"))) {
		MapName.RemoveAt(0, 9);
	}
	if (BeginWorld(MapName)) {
		InWorld.GetTimerManager().SetTimer(BeginPlayHandle, 
		  FTimerDelegate::CreateUObject(this, &ThisClass::BeginPlay), 1, false, 3.f);
	}
}

void UKopiTestWorldSubsystem::SetupBarista(AKopiBarista *Barista) const {
	const FTransform ZeroTransform = {};
	for (const int PlayerDriverId : PlayerDriverIds) {
		const TSubclassOf<UActorComponent> Class = GetPlayerDriverClass(PlayerDriverId);
		UKopiTestPlayerDriver *Driver = Cast<UKopiTestPlayerDriver>(
		  Barista->AddComponentByClass(Class, false, ZeroTransform, false));
		Driver->SetChaos(Chaos);
	}
}

This is the “easy” part; the core of the test functionality is in the UKopiTestHitFollowingPlayerDriver. Early in architecting KOPI, we decided to separate the “recipe model” and the tools that operate on the model. This way, the game “knows” that, for example, espresso with milk is made from espresso and milk; where espresso is the brew transformation of ground coffee beans. The coffee grinder and fridge advertise the ingredients given out (ground coffee beans and milk); the espresso machine equally advertises its transform operation (brew); the game can, given the state of the world, compute the sequence of steps necessary to make espresso with milk. In gameplay, we use this information to provide hints (here, take a cup).

Because of these hints, we can implement the UKopiTestHitFollowingPlayerDriver to do what a systematic and logical (even if a little slow) player would do. The difficulty in this is that this would only test the happy path scenario; and this is where the chaos parameter comes in. The higher the chaos, the more likely that the UKopiTestHitFollowingPlayerDriver will just “do stuff”: try to interact out of sequence, try to combine the ingredients in an unexpected way. During automated test run, we compute all combinations of interactions, and then run the tests until all interactions (whether allowed or not) have been tried at least once. This way, we can be sure that even if the players mash buttons, we won’t be caught unprepared.

When executing the test, this is how the test machines “play”.