Fast local iterations

In case you didn’t notice, we’re building a game. The game of course runs “without any bugs 🫣” in the Unreal Editor, but it is necessary to test the game on various local devices as quickly as possible. We could use our CI/CD, but that is a touch slower: it runs all the tests, it does all the work. Especially in tuning and debugging, it is useful to have a faster & isolated local build and release environment.

Here is how we are using the default WSL Ubuntu instance with:

Additionally, you will need a local-depot-configured Steam [Deck] client

Configure WSL

First, download Unreal Engine and Steamworks SDK, uncompress to /opt, make sure both directories are writable for the current user.

/opt$ ls -la
drwxr-xr-x  7 you root 4096 Jul 29 10:52 steamworks_sdk
drwxr-xr-x  6 you root 4096 Jun 24 15:09 ue5_4

Unreal Engine need C++ development tools and other packages, install

sudo apt install libatk1.0-0
sudo apt install libatk-bridge2.0-0
sudo apt install libxkbcommon-x11-0
sudo apt install libgbm-dev
sudo apt install libpangocairo-1.0-0
sudo apt install libasound2

With the unreal pre-requisites done, move on to install Powershell:

# Install pre-requisite packages.
sudo apt-get install -y wget apt-transport-https software-properties-common

# Get the version of Ubuntu
source /etc/os-release

# Download the Microsoft repository keys
wget -q https://packages.microsoft.com/config/ubuntu/$VERSION_ID/packages-microsoft-prod.deb

# Register the Microsoft repository keys
sudo dpkg -i packages-microsoft-prod.deb

# Delete the Microsoft repository keys file
rm packages-microsoft-prod.deb

# Update the list of packages after we added packages.microsoft.com
sudo apt-get update

# Install PowerShell
sudo apt-get install -y powershell

Next, install i386 packages for Steamworks SDK.

sudo dpkg --add-architecture i386

To host Steam Local Content Server, you will also need first install nginx.

sudo apt install nginx

After installing, configure the nginx by creating /etc/nginx/sites-available/steam with the following content

server {
    listen 80;
    listen [::]:80;
    root /var/www/steam;
    index  index.html index.htm;
    server_name _;

    client_max_body_size 2000M;
    autoindex on;
}

Then create symbolic link to this file in /etc/nginx/sites-enabled:

/etc/nginx/sites-available$ ln -s ../sites-available/steam steam

Make sure nginx is running sudo service nginx start.

Finally, create the working directories for build files and the content server in /var; these directories must be writable for your current user.

/var$
drwxr-xr-x  3 you root 4096 Jul 29 13:38 ue5_4
drwxr-xr-x  2 you root 4096 Jul 30 10:00 steam
drwxr-xr-x  3 you root 4096 Jul 29 13:02 www/steam

Verify prerequisites

If everything is installed as expected, these commands should work:

 # Powershell
 pwsh
 ... exit

# Steam C
/opt/steamworks_sdk/tools/ContentBuilder/builder_linux/steamcmd.sh 
... login <username> <password>
... < <steam-verify-code>
... exit

Configure WSL host

To allow Steam Deck [on your local network] to access the Steam depot on WSL, you will need to add port forwarding to the WSL instance. On the Windows host, run

~ > 
sudo netsh interface portproxy add v4tov4 
  listenport=80 
  listenaddress=0.0.0.0 
  connectport=80 
  connectaddress=(wsl hostname -I)

Additionally, make sure the firewall allows connections to port 80 [from non-public networks]. On another computer, verify that you can access the local depot.

Configure Steam [Deck]

For the Steam client to know where to look for your game, you’ll need to create the file steam_dev.cfg in the same directory that contains the Steam executable with the following content:

@LocalContentServer ...dreamonastick.com
  • Either the server IP or server hostname can be used.
  • If you need to specify another port then 80, it must be ip:port in quotes, e.g. "192.168.12.34:1234"
  • Don’t add http:// in front, HTTPS is not supported
  • Save this file, ensuring it is readable by the Steam user [on Steam OS, the deck user] in
    For Windows: C:\Program Files (x86)\Steam\
    For macOS: Steam.app/contents/macOS/
    For Linux or Steam OS: ~/.local/share/Steam/

Build

To build KOPI, we copy [or clone] its source code, then run

KopiNeo$
dos2unix ./Build.ps1 ; ./Build.ps1 -Target Development

Optionally replacing the -Target parameter with Shipping if desired. If you do not want to use up another Perforce workspace, and wish to treat the Linux source code as a read-only clone of the Windows directory, you can enable source code synchronisation with the Windows host by adding the file KopNeo/.wslsource with one line specifying the WSL host mount where the “source-of-truth” source code lives, for example

/mnt/d/Games/KopiNeo/

Then add -Sync parameter to Build.ps1 :

KopiNeo$
dos2unix ./Build.ps1 ; ./Build.ps1 -Sync -Target Development

Run build, verifying that the output indicates success

KopiNeo$
dos2unix ./Build.ps1 ; ./Build.ps1 -Sync -Target Development

...

##teamcity[message text='|[ |] UAT: |[RAW|] Stage command time: 20.39 s' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] ********** STAGE COMMAND COMPLETED **********' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] ********** PACKAGE COMMAND STARTED **********' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] Package command time: 0.00 s' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] ********** PACKAGE COMMAND COMPLETED **********' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] ********** ARCHIVE COMMAND STARTED **********' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] Archiving to /var/ue5_4' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] Archive command time: 6.98 s' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] ********** ARCHIVE COMMAND COMPLETED **********' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] BuildCookRun time: 82.20 s' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] BUILD SUCCESSFUL' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] AutomationTool executed for 0h 1m 22s' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[ |] UAT: |[RAW|] AutomationTool exiting with ExitCode=0 (Success)' status='NORMAL' timestamp='2024-07-30T09:58:30.000']
##teamcity[message text='|[*|] CI : Skipping tests in Development.' status='NORMAL' timestamp='0001-01-01T00:00:00.000']

The /var/ue5_4 directory should contain the newly built binaries:

/var/ue5_4$
du -h
873M    ./Linux/Kopi/Content/Paks
873M    ./Linux/Kopi/Content
536M    ./Linux/Kopi/Binaries/Linux
536M    ./Linux/Kopi/Binaries
1.4G    ./Linux/Kopi
260K    ./Linux/Engine/Content/Renderer
5.2M    ./Linux/Engine/Content/SlateDebug/Fonts
5.2M    ./Linux/Engine/Content/SlateDebug
5.5M    ./Linux/Engine/Content
188K    ./Linux/Engine/Extras/GPUDumpViewer
192K    ./Linux/Engine/Extras
2.0G    ./Linux/Engine/Binaries/ThirdParty/Vulkan/Linux
2.0G    ./Linux/Engine/Binaries/ThirdParty/Vulkan
23M     ./Linux/Engine/Binaries/ThirdParty/CEF3/Linux/Resources/locales
41M     ./Linux/Engine/Binaries/ThirdParty/CEF3/Linux/Resources
13M     ./Linux/Engine/Binaries/ThirdParty/CEF3/Linux/swiftshader
1.2G    ./Linux/Engine/Binaries/ThirdParty/CEF3/Linux
1.2G    ./Linux/Engine/Binaries/ThirdParty/CEF3
9.3M    ./Linux/Engine/Binaries/ThirdParty/MsQuic/v220/linux
9.3M    ./Linux/Engine/Binaries/ThirdParty/MsQuic/v220
9.3M    ./Linux/Engine/Binaries/ThirdParty/MsQuic
412K    ./Linux/Engine/Binaries/ThirdParty/Steamworks/Steamv157/x86_64-unknown-linux-gnu
416K    ./Linux/Engine/Binaries/ThirdParty/Steamworks/Steamv157
420K    ./Linux/Engine/Binaries/ThirdParty/Steamworks
3.2G    ./Linux/Engine/Binaries/ThirdParty
9.6M    ./Linux/Engine/Binaries/Linux
3.2G    ./Linux/Engine/Binaries
3.2G    ./Linux/Engine
4.6G    ./Linux

Publish

Use the Steamworks SDK to publish the build to the local depot. First make sure that your Steamworks user has the privileges to publish and edit metadata, then run

$ /opt/steamworks_sdk/tools/ContentBuilder/builder_linux/steamcmd.sh 
Steam> login user password
... Steam Verify CODE: ABC123

Steam> run_app_build -preview ~/Games/KopiNeo/Platforms/Steam/app_build_2909960.vdf

Optionally replacing ~/Games/KopiNeo with the directory where you cloned or copied the KopiNeo source code to.

Once completed, you should see the new build appearing at https://partner.steamgames.com/apps/builds/$app-id, and the Steam clients should automatically download the latest version [from the local depot].

Common Traps

  • Steam ContentBuilder fails with “ERROR”:
    – verify that the depot and app IDs are correct and verify that changes are published on Steamworks.
  • Steam Deck fails to download the newly published game:
    – Verify that the steam_dev.cfg is in the correct location and readable by the deck user.
    – Verify that nginx is serving the right content and that it is accessible from another computer; if not, verify netsh interface portproxy show all and verify firewall allows incoming connection
    – Restart Steam Deck

Leave a Reply