Progress Update: Game Engine Beta v0.0.9

The last update I gave on the game engine was back in Nov 2017. A lot of improvement has occurred since then. I fixed over 15 bugs and enhanced the Collision Detection System and the Digital Asset Loader. Below is a video showing beta-version v0.0.9 of the game engine.

 
 

Improving the GJK

I have mentioned before that the collision detection system is the most complicated aspect of a game engine. Interestingly, the algorithm is quite simple. However, the complication arises from the several boundary conditions that you need to account.

The Collision Detection system worked fine between objects of similar size. For disproportionate size objects, i.e., terrain and a cube, the collision algorithm would fail 50% of the time.

There were instances where detection happened way too early or too late. Other cases were even weirder. For example, merely changing the location of the cube would cause the detection to fail. I recall an instance where changing the orientation of the cube would cause a failure.

I recall spending most of my Thanksgiving and Christmas holiday working on the issue stated above. It took me over two months to improve the collision detection between disproportionate objects. It is not perfect, but it works within constraints.

What I learned from these two months working on improving the engine, is that the GJK algorithm has several boundary conditions that your code needs to account. In the previous version of the engine, I was not considering these boundary conditions; thus the reason for the early or late detections.

If you find yourself in this situation, I recommed to take a look at these books:

Improving the Digital Asset Loader

I also improved the Digital Asset Loader (DAL). If you don't know what a DAL is, it is merely a script that loads 3D models into the engine. Previously, the script would output rendering data directly to a terminal. I would copy and paste this data into an engine file.

I improved the script to write directly to the engine file. Not only that, but you can now choose what you want to import. i.e., just 3D models, 3D animations, light, or a combination of 3D models with animations.

Books I had to read to develop a game engine

If you have decided to develop your game engine, you may be wondering where to start, what books to start reading, etc. If you are in this situation, I recommend getting a copy of the following books:

Books to get started

Math Engine

  1. 3D Math Primer For Graphics and Game Development

Rendering Engine

  1. OpenGL Superbible: Comprehensive Tutorial and Reference
  2. Graphics Shaders: Theory and Practice, Second Edition

Physics Engine

  1. Physics for Game Developers: Science, math, and code for realistic effects
  2. Game Physics Engine Development: How to Build a Robust Commercial-Grade Physics Engine for your Game
  3. Real-Time Collision Detection

Out of all the books I've read, they are the best of the best, so I strongly recommend them.

Now, if you want a comprehensive list, the list below should help you.

Books to develop the Math Engine

  1. Mathematics for 3D Game Programming and Computer Graphics
  2. Quaternions and Rotation Sequences: A Primer with Applications to Orbits, Aerospace and Virtual Reality
  3. Quaternions for Computer Graphics
  4. 3D Math Primer For Graphics and Game Development
  5. Practical Linear Algebra: A Geometry Toolbox
  6. Visualizing Quaternions
  7. Curves and Surfaces for Computer Graphics
  8. Essential Mathematics for Games and Interactive Applications: A Programmer's Guide

Books to develop the Rendering Engine

  1. OpenGL 4.0 Shading Language Cookbook
  2. OpenGL Insights
  3. Real-Time Rendering, Third Edition
  4. OpenGL Shading Language
  5. Foundations of 3D Computer Graphics
  6. Texturing and Modeling, Second Edition: A Procedural Approach
  7. OpenGL Superbible: Comprehensive Tutorial and Reference
  8. Graphics Shaders: Theory and Practice, Second Edition
  9. Graphics Programming Methods
  10. Real-Time Shadows

Books to develop the Physics Engine

  1. Physics for Game Developers: Science, math, and code for realistic effects
  2. Numerical Methods for Scientists and Engineers
  3. A Student's Guide to Vectors and Tensors
  4. Real-Time Collision Detection
  5. Game Physics Engine Development: How to Build a Robust Commercial-Grade Physics Engine for your Game
  6. Computational Geometry: Algorithms and Applications
  7. Game Physics Pearls
  8. Geometry for Computer Graphics: Formulae, Examples and Proofs
  9. Geometric Tools for Computer Graphics
  10. Computational Geometry in C
  11. Collision Detection in Interactive 3D Environments
  12. Game Physics

Learning 3D Modeling tools (Blender)

If you are wondering why you need to learn Blender, read the following article.

  1. Blender Master Class: A Hands-On Guide to Modeling, Sculpting, Materials, and Rendering
  2. Blender Foundations: The Essential Guide to Learning Blender 2.6
  3. Learning Blender: A Hands-On Guide to Creating 3D Animated Characters
  4. Python 3 Object Oriented Programming- Blender provides an API which allows you to extract 3D modeling information. To use the API, you need to know a bit of Python.
  5. Digital Modeling

Game Engine Architecture

  1. Head First Design Patterns
  2. Game Programming Gems
  3. Best of Game Programming Gems
  4. 3D Game Engine Architecture
  5. 3D Game Engine Design: A Practical Approach
  6. C++ Common Knowledge: Essential Intermediate Programming
  7. C++ Pointers and Dynamic Memory Management
  8. Effective C++: 55 Specific Ways to Improve Your Programs and Designs
  9. C++ Templates: The Complete Guide
  10. Design Patterns: Elements of Reusable Object-Oriented Software
  11. The C++ Standard Library: A Tutorial and Reference
  12. Thinking in C++, Vol. 1: Introduction to Standard C++, 2nd Edition
  13. Thinking in C++, Volume 2: Practical Programming
  14. Game Programming Algorithms and Techniques
  15. API Design for C++
  16. Game Programming Patterns

Hope it helps.

The mistake I made as a game engine developer

I have made several mistakes as a game engine developer. However, there is a significant mistake I did in 2017 which I want to share with you.

At the start of 2017, I decided to put the game engine (Beta v0.0.4) to the test by developing a soccer video game. I wanted to test the limits of the engine, and a soccer game allowed me to do so. For instance, a soccer video game makes use of 3D animations, collision detection, rendering efficiency, etc.

At first, developing the game was fun. I learned about Artificial Intelligence, Graph Theory, how to develop a messaging system among the players, Dijkstra's algorithm, etc.

Here are some videos that show the evolution of the game:

First I learned how to control the armature of a 3D model as it is animated. In this case, managing the kick animation as it is about to kick the ball. If you are interested how I did this, read this article.

 
 

Then I learned how to enable communication among the players. That is, I implemented an AI messaging system:

 
 

I then improved the AI to analyze defending strategies:

 
 

Improved the AI to take into account all 22 players on the field:

 
 

I also cleaned up the AI architecture:

 
 

Finally implemented a pathfinder algorithm:

 
 

Implementing a video soccer game was fun but also time-consuming. In particular, creating the dozens of animations required for smooth gameplay. For example, you need to create a kick, passing, running, defending, halting animations, etc.

Developing the AI for the game took a lot of time. The AI is by far the most complicated aspect of a soccer game, and I spent a lot of time implementing it.

However, six months after I started working on the game, I stopped. I stopped development once I realized that I had lost focus. My goal has always been to develop a 3D game engine. I thought that developing a soccer game will help the engine. However, it didn't. Instead, I lost six months of development.

I learned a valuable lesson:

Do not test the engine with games that require complicated Artificial Intelligence. Instead, develop simple games that strategically test the engine's efficiency and response.

If you want to test the limits of your engine through complicated games, go ahead. But if you are an indie-developer like me, keep in mind that you have limited resources and that you have to be strategic about your time.

Ever since then, I've been focused on the engine again.

In version Beta-v0.0.5, I ported the engine to use the Metal API. (Since I already had the soccer game, it seemed convenient to use it for testing.)

 
 

I improved the rendering efficiency of the engine by implementing Frustum culling with a BVH tree algorithm (Beta-v0.0.6).

 
 

In version Beta-v0.0.7, I implemented a particle system, which can generate smoke, explosions and other particle types.

 
 

In version Beta v0.0.8, I fixed memory leakages and improved the rendering-ordering algorithm.

 
 

In the upcoming version, which I hope to share with you in a couple of weeks, I significantly improved the collision detection system of the engine.

As indie developers, we have limited resources and its crucial that you are strategic about them. I lost six months of development for not analyzing the impact a complicated AI game will have on the engine. I hope you don't make the same mistake as I did.

Thanks for reading.

Should you create a game engine or use an existing engine?

If computer graphics, mathematics, and programming fascinate you, then Game Engine Development may be suitable for you. The only downside is that it takes a long time to develop a game engine. In my experience about three years.

However, if you want to have a game prototype within six months, then use a game engine such as Unity or Unreal instead.

I say this from experience. Back in 2013, I decided to develop a 3D game engine from scratch. Three years later, in 2016, I was able to complete the "basic" framework of the engine.

Throughout these three years, I had to learn computer graphics, OpenGL, linear algebra, design patterns, computational geometry algorithms, etc. I had to read a lot. If you don't believe me, here is a photo of all the books I had to read to develop the "basic" framework of the engine.

1476242358374.jpeg

Notice that I keep mentioning: "Basic Framework of the engine." Such statement is vital to keep in mind. After three years of development, all I had was a skeleton; the engine could only run simple games. And it lacked many, many features.

For most of last year (2017), I focused on improving the engine; I fixed hundreds of bugs and implemented features. The engine has improved a lot. However, it is not ready for primetime yet. My goal for this year (2018) is to make further improvements. Hopefully, version 1.0 will be ready by the end of 2018.

Here is beta version v0.0.8 of the engine as of today

 
 

Why do I tell you this? So that you understand the work required to develop a game engine. It is not an easy task, and it will take a lot of time.

So, if you want to have something ready within a reasonable amount of time, use a game engine to develop your game. But if you are interested in learning Computer Graphics, OpenGL/Metal, Design Patterns, Computational Geometry Algorithms, Linear Algebra, Object-Oriented Programming concepts, and don't mind going crazy, then go ahead and develop a game engine.

How long does it take to develop a game engine?

I've been getting several emails in the last couple of months from indie developers and college students interested in developing a game engine. The bulk of these emails revolves around the question: "How long does it take to develop a game engine"?

I don't know how long it will take you, but I can tell you how long it took me to go from a blank file to this:

 
 

In the beginning

When I decided to develop a 3D game engine, I knew next to nothing about C++, Design Patterns, and algorithms. I knew C programming, but I didn't feel comfortable since most of what I knew about programming was self-taught. My bachelor was in Bio-Engineering, not Computer Science. Thus, you can safely assume that my programming skills were borderline basic.

I also knew nothing about OpenGL, Shaders and Computer Graphics.

Developing a game engine required that I mastered all these concepts: C++, Design Patterns, OpenGL, Computer Graphics and Linear Algebra. So, for the first seven to eight months (Feb 2013-Aug-2013), I decided to dwell deep into these concepts. I read a lot. I spent my hours before heading to work, after work and weekends, just studying, reading and learning as much as I could.

My first attempt

It was around Aug-2013 that I started coding the game engine. Around this time I was not wholly knowledgeable about C++, OpenGL nor Linear Algebra but I decided to start coding the engine nonetheless. Unfortunately, the engine's architecture was a total mess, and by Nov 2013, I had to start all over again.

It was not until a year later (Feb-2014) that I succeeded in rendering the first 3D character on the screen.

A lot of work with nothing to show

The year 2014 was depressing. I was spending all of my free time working on the engine. I kept reading and reading about OpenGL and Computer Graphics. However, most of what I implemented would fail. I must say that it was also a year I learned a lot.

From Feb 2014 to May 2014, I worked on the Rendering Engine and added support for 3D model rendering, textures, normal maps, and multi-image. From May 2014 to July 2014, I implemented a crude Digital Asset Imported. That is, I was able to export a 3D model from "Blender" and import it into the engine.

The rest of 2014 was spent on implementing a scenegraph, fine-tuning the Digital Asset Imported, adding support for 3D animations, and learning how to use Blender 3D.

 

Testing a walking animation in the engine. 

 

You may not know this yet, but you will need to wear your artistic hat if you want to develop a game engine. It is crucial that you learn how to model 3D characters using tools such as Blender and the workflow used by 3D artists. If you do not take time to learn this, you will not be able to import the correct data into the engine from Blender.

Finally, something to show

It was not until April 2015 that the Rendering Engine showed signs that it was working. On April, I was able to complete Beta version v0.0.1 of the engine.

 
Version v0.0.1 The game engine is capable of rendering 3D models with textures/normal maps, animation, skyboxes, fonts, sprites and contains a digital asset importer.
 

The engine had support for 3D models, normal maps, buttons, joystick, shadows and 3D animations. However, most of what I had implemented was crude. Remember, I was learning and failing hard along the way.

The complicated part of an engine

On April 2015, I started working on the Physics Engine. The physics engine, along with the Collision Detection System, is the hardest part of a game engine. Trust me.

The year 2015, was also the year when my temptations to quit development was the strongest. I can't even explain how terrible it is to work on a Collision Detection System. I would work until 2 am almost every day, and every time, that damn Collision System would fail. It was not until October that I finally was able to get the engine to detect collision among convex objects. However, there was a problem with collision tunneling. Essentially, colliding objects would penetrate each other. Unbeknown to me, it was going to take me over a year and three months to get the collision system working.

I Finally Did it

It was not until July 21, 2016, around 2 am, that I succeeded in developing the basic framework of the game engine. Trust me, when I saw the engine working, I screamed, I jumped, and I started dancing Salsa (I'm Latino. Salsa music is in my blood).

 
 

It took me a little bit more than three years to develop the basic structure of the game engine. And I have to emphasize the word "basic.". The engine had a basic structure, but it was not ready for prime-time. There was a lot more work to do.

The First Demo

I knew the engine needed a lot more functionality. However, I couldn't wait to develop a simple demo. So, a month later, I developed the demo shown below:

 
 

Development started to pick up

After the first demo, development started to go by faster. By October 2016, I had improved the collision detection and fixed issues with 3D animations.

 
In this beta version v0.0.3 of the engine, animations and collision detection can work simultaneously. The BHV algorithm was improved helping the engine make better decisions when pairing up 3D models for collision detection. The MVC (Model-View-Controller) flow of information was also improved.
 

By Dec 2016, I developed the second demo which focused on showing the collision-detection features of the engine:

 
 

By Jan 2017 I implemented a very primitive particle system. The particle system was so rudimentary that I completely rewrote it later on.

 
In this beta version v0.0.4, I implemented a primitive particle system, thus allowing explosion effects to occur once a missile hits the asteroid. I also enabled multi-touch, this allows the spaceship to turn as it speeds up. I also set up collision filters among object types. For example, object A and object B can collide; object A and object C can collide, but any collision among object B and Object C is ignored.
 

A huge learning moment

From Jan 2017 to Jun 2017, I wasted my time developing a soccer game. The idea was to create a soccer game so that I could improve the engine. However, I soon found myself spending more time coding the gameplay, rather than spending time fixing other issues in the engine.

 
 

And this was a huge learning moment for me. I realized that I should not waste my time coding a gameplay. And more importantly, that each game I develop, must serve a purpose towards the improvement/testing of the engine.

Porting to Metal

By Sept 2017, the game engine got a total makeover. From Jun 2017 to Sept 2017, I worked on porting the engine from OpenGL ES to Metal, Apple's Graphics API. The porting was not hard at all; I have to say that Metal is a lot nicer and cleaner to work with than OpenGL. I was surprised that I ported the engine in less than four months.

 
This video shows the game engine using the Metal API for its rendering operations. The game engine no longer uses OpenGL.
 

Since I already had the soccer game demo, I used the same game (with different 3D models) to test the engine with Metal.

Why render what you can't see

By Oct 2017, I worked on a feature that I should have worked on years ago. For most of the development, I had put Frustum Culling aside. The idea of Frustum Culling is that there is no point for the engine to render entities that the camera can't see. Implementing this feature improved the engine's FPS (frames-Per-Second).

 
Improved the rendering efficiency of the game engine by implemented Frustum Culling + BVH tree algorithm.
 

A true Particle System

Earlier, I had mentioned that I developed a primitive particle system. Well, by the end of Oct 2017, the time had come for me to implement a real particle system. I had fun working on this feature.

 
Added particle system to the engine. The engine can generate smoke, explosions and other particle types.
 

Memory Leakage is not acceptable

And finally, by Nov 2017, I fixed several memory-leakages in the engine and a rendering ordering issue with the engine.

 
In version v0.0.8 of the game engine I fixed some memory leakage issues and improved the rendering-ordering algorithm.
 

In Conclusion

I've been working on my game engine for more than four years now. It took three years to develop a basic framework. And an additional year to improve the engine. And I'm still not done with the engine.

I hope that this post can give you a sense of what awaits you if you decide to develop your game engine.

Thanks for reading.