If you have decided to develop your game engine, you may be wondering where to start, what books to start reading, etc. If you are in this situation, I recommend getting a copy of the following books:

Books to get started

Math Engine

1. 3D Math Primer For Graphics and Game Development

Rendering Engine

7. OpenGL Superbible: Comprehensive Tutorial and Reference
8. Graphics Shaders: Theory and Practice, Second Edition

Physics Engine

1. Physics for Game Developers: Science, math, and code for realistic effects
2. Game Physics Engine Development: How to Build a Robust Commercial-Grade Physics Engine for your Game
3. Real-Time Collision Detection

Out of all the books I've read, they are the best of the best, so I strongly recommend them.

Now, if you want a comprehensive list, the list below should help you.

Books to develop the Math Engine

1. Mathematics for 3D Game Programming and Computer Graphics
2. Quaternions and Rotation Sequences: A Primer with Applications to Orbits, Aerospace and Virtual Reality
3. Quaternions for Computer Graphics
4. 3D Math Primer For Graphics and Game Development
5. Practical Linear Algebra: A Geometry Toolbox
6. Visualizing Quaternions
7. Curves and Surfaces for Computer Graphics
8. Essential Mathematics for Games and Interactive Applications: A Programmer's Guide

Books to develop the Rendering Engine

1. OpenGL 4.0 Shading Language Cookbook
2. OpenGL Insights
3. Real-Time Rendering, Third Edition
4. OpenGL Shading Language
5. Foundations of 3D Computer Graphics
6. Texturing and Modeling, Second Edition: A Procedural Approach
7. OpenGL Superbible: Comprehensive Tutorial and Reference
8. Graphics Shaders: Theory and Practice, Second Edition
9. Graphics Programming Methods
10. Real-Time Shadows

Books to develop the Physics Engine

1. Physics for Game Developers: Science, math, and code for realistic effects
2. Numerical Methods for Scientists and Engineers
3. A Student's Guide to Vectors and Tensors
4. Real-Time Collision Detection
5. Game Physics Engine Development: How to Build a Robust Commercial-Grade Physics Engine for your Game
6. Computational Geometry: Algorithms and Applications
7. Game Physics Pearls
8. Geometry for Computer Graphics: Formulae, Examples and Proofs
9. Geometric Tools for Computer Graphics
10. Computational Geometry in C
11. Collision Detection in Interactive 3D Environments
12. Game Physics

Learning 3D Modeling tools (Blender)

If you are wondering why you need to learn Blender, read the following article.

1. Blender Master Class: A Hands-On Guide to Modeling, Sculpting, Materials, and Rendering
2. Blender Foundations: The Essential Guide to Learning Blender 2.6
3. Learning Blender: A Hands-On Guide to Creating 3D Animated Characters
4. Python 3 Object Oriented Programming- Blender provides an API which allows you to extract 3D modeling information. To use the API, you need to know a bit of Python.
5. Digital Modeling

Game Engine Architecture

1. Head First Design Patterns
2. Game Programming Gems
3. Best of Game Programming Gems
4. 3D Game Engine Architecture
5. 3D Game Engine Design: A Practical Approach
6. C++ Common Knowledge: Essential Intermediate Programming
7. C++ Pointers and Dynamic Memory Management
8. Effective C++: 55 Specific Ways to Improve Your Programs and Designs
9. C++ Templates: The Complete Guide
10. Design Patterns: Elements of Reusable Object-Oriented Software
11. The C++ Standard Library: A Tutorial and Reference
12. Thinking in C++, Vol. 1: Introduction to Standard C++, 2nd Edition
13. Thinking in C++, Volume 2: Practical Programming
14. Game Programming Algorithms and Techniques
15. API Design for C++
16. Game Programming Patterns

Hope it helps.