Getting Started
Before you begin
Make concrete plans
Like any software venture, the first step in making a game engine is to plan. The first thing you should do - especially if this is your first game - is to have a concrete idea of what your game will be like.
It may be that you decided to jump into this without having any of those settled. While I applaud you enthusiasm, you should really stop and think about them. For those readers, I have the following tips
Consider the Workflow
Writing an engine is not even half of the difficulty of making a full-scale retail video game. You should really consider what steps you’ll want to take to generate the actual gameplay beyond the engine itself before you even start writing it. It’s very important to keep in mind which tools you can bring in the fold to make the game development easier and the possibility of bringing other people into your team and having them take on some of the work. Think about the data structures you’re going to need to input into your engine and how those data structures will be put together.
Let’s look at levels for an example. You could make the level for your game one big bitmap graphic that just gets drawn directly to the screen. And if you do that, you could manually write boundries and colliders and positioning for objects on the screen. Or you could load your picture into Inkscape to draw vectors, edit the classes of each path you draw and then import them into your engine as data for it to parse. Or you could use a map editor like Tiled which lets you assign properties to tiles and place objects visually so your level designers can do most of their job in one place. All three of these are valid workflows, and they have their advantages, but I’m sure with a little bit of imagination you could see how all of them have their own disadvantages as well.
Abandon the third dimension
3D games increase the complexity of a game engine exponentially. 2D math is so simple, relatively speaking, that you almost don’t have to worry about optimization in the modern age with current hardware. You can make countless mistakes and inefficencies before you start to see performance start to drop. 3D games require an absurd amount of work, and people tend to be extra picky about 3D graphics. New 3D engines are rare for a reason, and they’re one of the reasons why companies like Epic and Unity Technologies managed to become so huge - it makes a whole lot more business sense to just buy in than to make your own engine, even if you do have talented staff that could make their own.
Beyond that, making a 3D engine requires a tremendous amount of math and theory, and getting everything right can be incredibly tedious. In contrast, if you took geometry and triginometry in high school, 2D engines should be completely within your grasp.
Some frameworks do have code to support 3D rendering, but I would suggest to avoid the temptation. If you must make a 3D game, I would suggest using pseudo-3D instead, but this book is supposed to be for first timers: just stick to 2D!
Pick good tools
I briefly talked about tooling earlier, but tools are important. I’m a believer in FOSS, but when it comes to tooling, the single most important factor to you is how well those tools serve you. For your graphics you could use Gimp and Krita or Inkscape if you prefer to work in vectors, but some of the most common digital art tools in industry are Adobe Photoshop and Illustrator. The tool I would generally recommend to you if you are using pixel art specifically is Aseprite, which offers important features like layers, tilemaps, and animation tools. I had also mentioned Tiled, which will let you take those tilemaps you generated in Aseprite and make them into full-blown maps that you can populate with objects and data from one convenient program.
But by far the most important tool you will need to pick are the frameworks and libraries you will build your engine with.
Frankly, I cannot give you good advice when it comes to what you should pick, because the choices you make will depend on your confidence with your skills and the requirements of your game. As you might guess, I prefer LÖVE, but nothing is perfect and LÖVE is not an exception. One big drawback of LÖVE is that if you don’t program your game entirely in Lua, you will be in charge of compiling versions of it for each platform you want to suport. That means that if you want to avoid that you will also have to avoid libraries that have components written with C, which means you can’t use libraries like LPeg or most LuaRocks. The same would be true if you were to build on the PyGame framework, but it seems that most Python libraries are written in pure Python and the ecosystem is so large you’re bound to find an alternative pure Python library. Alternatively you could build your game on a web browser instead and it’s trivial to get it running on just about everything without necessarily needing an extra framework to support what you’re doing.
Of course, you’ll also want to have good software writing tools, like an IDE. While the rest of the world might be using Visual Studio Code, I might recommend you instead to try [VSCodium], the all-FOSS version with Microsoft’s tracking disabled. Regardless, you’ll want an IDE with powerful debugging tools, because the nature of developing interactive software means many tightly interlocked components where many things can go wrong. Something that can stop the machine and inspect it’s state is an invaluable tool.
Set up a versioning system
And make sure you know how to use it, too. This is especially important when working with other people.
If you’re already familliar with Git, you’re probably already good. Though, do keep in mind that Git is only really good with text files. When it comes to the point that you’re working on assets for the game, you might want to start using a versioning system thats better with binary files. I don’t have any advice about which one to use at the moment, but I’ve heard good things about mercurial
Take your time and make it fun
There’s no need to rush. The game will be done when it’s done. When you consider options, take the time to really consider them. Keep in mind that there are some parts of the plans you are making at this phase that will not happen the way you wanted to. Some decisions will be very hard to come back from when you’re further down the line, so it’s important to thouroughly consider all aspects of each choice.
At the same time, don’t stress too much with these decisions. To quote Robert Burns:
The best laid plans of mice and men often go awry.
It’s impossible to see the future. Your plans will change. You might find that the whole project never materializes. It’s way too early to start worrying about failure when you haven’t even started yet! Remember that the whole point of a game is to have fun, and writing a game engine should be fun too.