Bird’s Eye View

Goal

Empower the reader to see the big picture of a complete web app within a short document.

Summary

Through building a complete working example I will talk about the concepts involved in, and over-all architecture of, developing a web app. This will help you formulate what areas you would like to study in greater detail and help you make use of that study by putting it into context. To help with this I will provide a library of resources on each topic.

Motivation

Just the ability to see the outline of this document is something I would have really appreciated when I was trying to fit the pieces together, however to provide the details of any specific piece, I could never do as good of a job as a search engine.

There are many ways to find the syntax of a method in a programming language, or how to solve a problem in that language if given a starting environment, or even something more complex like trouble shooting a task within a piece of software. However, what I found when I started learning programming was that it is not easy to find a good explanation of the big picture, of how everything fits together. Without this you are unable to build anything meaningful, because there is a continual search for details that have no context.

The aim of this website is to provide that explanation in plain English, defining terms only where necessary, to the extent that someone with no prior programming experience will be able to know what details to look for and where to find them by looking at a short document. The end goal is the ability to be able to build an elementary version of a complete web application and understand how to be a software developer. At the end there will be suggestions for what directions to take for more advanced learning.

Rather than make you read so many details that you get confused and tired before understanding the underlying concepts. I want you to be able to see the bare minimum that allows you to go exploring on your own. One great way to think of this document is a collection of keywords displayed in context for you to search with and find the information you need. I am thinking about what you need to get things up and running, my goal is not to document every fact that I have.

A Technology Stack is the set of software tools that you use to develop your web application. My running example will be the MEAN technology stack (Mongo Express Angular and Node), I will also be talking about other alternatives along the way. In general the principles learned from creating a web app with one stack transfer to another.

I am going to talk about the main categories of software tools to consider, without adding unnecessary complexity. Meaning that there is core functionality that your app must have in order to run and then there is always another layer of complexity if you are ready for it.

Table Of Contents

  1. Operating system and shell
  2. Text Editor
  3. Database software
  4. Server Side
    1. Language
    2. Software
    3. Framework
  5. Client Side
    1. Language
    2. Framework
  6. Web Hosting
  7. Versioning Hosting
  8. Domain Name Registry

Section 1: The Web App Operating System And Shell

A CLI (Command Line Interface) is an interface that allows the user to Interact with the computer by typing out commands. A GUI (Graphical User Interface) is an interface that allows the user to interact with the computer by manipulating graphics. A Shell is any software tool that allows the user to interact with the operating system, most operating systems have a GUI option and a CLI option. It would be a good exercise for you to figure out what the default CLI of your operating system is, search for a list of commands, and then experiment with them just to get the general idea. Currently I do my work on a Windows machine, and then upload to a Linux remote server (I talk more about this in the web hosting and version hosting sections).

The most popular tool for programmers to communicate with the operating system is the bash Shell which is a CLI, git has a tool called gitbash which can be run on Windows because windows CLI is significantly different than bash which is found on linux and mac machines. The basic commands that are required for using bash are listed below, these all have their equivalents in other CLIs. It would be good to experiment by searching for “command line equivalents” and seeing what lists come up.

In the commands below when I say “directory”, this is the path to a location or file. Current working directory is the directory you are currently in. It’s also good to know that if you see the formatting, “<something>” this signifies where you as a user would type your own input.

  • $ pwd – prints your current working directory
  • $ ls – prints a list of the files that are in your current directory
  • $ cd <directory> – changes your current directory
  • $ mkdir <name> – creates a new directory within your current directory
  • $ rm <file name> – deletes a file
  • $ rm –r <directory> – deletes a directory
  • $ cat <file> – prints the contents of the file
  • $ nano <file> – opens a file in the nano text editor (you must have the nano text editor for this to work, if you don’t google what text editor your operating system does have, and the command line commands for it).

Popular Alternatives to a Unix based Operating System (OS) such as Linux are Mac and Windows. Bash is by far the most popular CLI but there is also a great use for something called GitBash. Gitbash is especially helpful if you are using windows, because Window’s default CLI has significantly different commands than bash, and Gitbash is easily installed on Windows.

The reason it is important to talk about your operating system, and especially the CLI that you use is because a lot of times what you will want to do as a programmer is done best through a CLI, or is only done through a CLI. For example, when you connect to your remote host server in order to update your web app, this will probably be the case.

Section 2: The Web App Text Editor

An Integrated Development Environment (IDE) is a set of software tools, centered around a text editor, that all work together to allow a developer to write code. The Environment can include many things, examples are code completion (this is like spellcheck but for coding), formatting (which keeps your code looking pretty), and a CLI for typing commands. The most simple tool you would use in order to manipulate a code file, which are just text files, is a text editor, however my running example for our web app will be using the IDE Microsoft Visual Studio Code. This is not to be mistaken with Microsoft Visual Studio, I will not be going into the difference between them except to say that MS Visual Studio Code is smaller than MS Visual Studio. They both have free versions if your curious to explore and plenty of documentation through Microsoft. Popular Alternatives to Visual Studio Code are Sublime Text, and Atom.

I will talk more about programming languages in the client side framework and server side framework sections later. For now I just want to say that I will be focused on javascript for my example web app. Once you have installed Visual Studio Code Look at the “extensions” button on the left hand side, here is where you can add more functionality to the IDE. This is not something I will be talking about because the functionality I will make the most use of, javascript code completion, is already part of VS Code. To see what code completion is create a new file named <something>.js (the .js is important because this is what tells VS Code to treat your code as javascript) and begin typing any javascript code. For example type Math.floor slowly and with each keystroke you should see Visual Studio giving you suggestions of the javascript code that it thinks you want.

Section 3: The Web App Database Software

My running example for database software will be MongoDB, this is a noSQL database, I will talk more about this later, but the only other popular alternatives would be a SQL database, such as MySQL.

MongoDB allows you to create a relational database this means that the documents in the database, whether it’s SQL or not have the ability to be linked together, by unique ID’s. Relational databases are important Because they prevent the unnecessary duplication of information. For example, if you are storing an “office” document in your database, rather than including all of the “employee” information in that same document you would create a different document for each employee, each of those documents would contain a unique id, this id is what you would add to your “office” document. This employee id could also be added anywhere else you wanted to refer to an employee.

The two main types of databases are SQL and “NoSQL”. SQL stores data in tables rows and columns, these tables are linked together, making the database relational. NoSQL is very similar in big concept but different mainly in syntax. NoSQL stores JSON objects instead of tables, which are key value pairs that, as their value, can have anything, an array a string a number or even another object. I like working with JSON because I find it to be very powerful and flexible, but best of all is that you can use the same javascript syntax throughout your whole technology stack, (I will talk more about this in server side frameworks and client side frameworks). It would be good to download and install MongoDB, they have great instructions on their website, and there is a free “Community” version.

Once downloaded you will need to create a directory in your root directory called “data” and inside that directory you will need to create another directory called “db”. MongoDB needs this to function. A root directory is the parent directory of all other directories, you can get to this on the command line by typing the following.

$ cd /

You will not need to run mongodb in auth mode just to figure out how to get a complete app up and running, however you will eventually need to start thinking about security. When the time comes you will need to look at mongoDB’s documentation to find how to create a user, and then run mongoDB in auth mode and connect to it with that username and password. If you don’t do this, hackers can connect to your database and act on it, this is no big deal if all you have in there is the fake data that you may have been testing your app with, but make sure to look into this before you actually put your app to use.

If you have installed mongodb you can run it by typing the following.

$ mongod

Once it is running without error, and says “listening on port 27017″ (default port, it can be something else) if you open a different CLI window, you can also run the following command.

$ mongo

This will open the mongo shell, which is used for connecting to and interacting with the running instance of the database. You may need to be in the correct directory to run these commands, which is probably (depending on where you installed it, “/program files/mongodb” and then keep going until you see the “bin” folder, inside there should be mongod.exe and mongo.exe, once you are in the same directory as these files the commands will work.

Go ahead and search for mongo shell commands and experiment with adding and deleting data from the database. Remember that the mongoDB official documentation is a great place to look. Also, if you have created a user and figured out how to run mongoDB in auth mode try to use the mongo shell to log in as that user.

Keep in mind that the way the server side code that we are going to create for our web app will interact with the database is not so different from the way you are already interacting with it using the mongo shell.

Section 4: The Web App Server side

In a web app there is the server side and the client side. While browsing the internet, if you attempt to connect to a website via a link or by typing the address into the address bar. You are the client, asking the server to send you the client side code. Once you receive the client side code for a particular web app, as you run the code on your machine, it will send requests to it’s server for further information or to update information that you have given it (you will be writing these requests, I will talk about them in the client-side section).

HTTP means Hyper Text Transfer Protocol. What is most important to you about this is that the client-side code of your web app will have methods to send an HTTP message and receive a response (this is how you will talk to your server-side code). And the server-side code of your web app will have methods to receive and respond to HTTP requests (this is how you will know what the client-side code wants from the database and be able to send that information back).

The server houses both the database, and software that “serves” your web app, meaning it handles requests that come to it from clients by responding to them. Also on the server, to begin with is the client side code, once a request comes from someone trying to use your web app the server will send the client side code to them. Then, while your web app is running on their computer the server-side code and client side code that you have written will interact with each other by HTTP. Similar to this, just as you have been interacting with the MongoDB database through the mongo shell, your server side code will interact with the database. In this way the database, the server-side code, and the client side code are the three separate and communicating parts of your web app while it is in use by someone.

Server Side Language

First you want to pick a programming language to work with, then see what your options are in that language for server software and frameworks, this is going to be the same process for client-side. My running example will be using Javascript, some popular alternatives for server side languages are Java, C# (Microsoft’s .NET), PHP (WordPress). Javascript is actually the least popular server side language of this group. However, it is only recently an option and it is used everywhere, for many purposes, and is only becoming more popular. Also, I like coding with it, I like the syntax, and the way it’s designed, so I jump at any opportunity to use it. for client-side, it is basically the only choice, so how great that you can now use it server-side as well! It would be a great idea to find a website that gives you coding problems in javascript, and a beginner’s tutorial. A good one is:

freecodecamp.com

Server Side Software

My running example for server software will be Node.js. It is because of Node.js that we are able to use javascript on the server.

Download and install Node.js, it is free and they have a great website. Node should also come with Node Package Manager (NPM) I am not going to talk too much about this except to say that it is a very useful tool to download whatever additional software you may need and to automatically have it saved to your projects dependency list. To make sure you have successfully installed both of them, in your CLI type the following.

$ node –v

$ npm –v

If these commands show you the version of the software then you have successfully installed them.

The next thing you will be doing with node is only after you have written your server code. Then, to run it (to start your server) you will type the following.

$ node <your file>.js

I will talk more about actually writing the server file when I talk about Express in the server-side framework section.

Server Side Framework

My running example for a server side framework is Express.js. Express is by far the most popular framework for Node.js, but there is also, Koa, Meteor, and many others, many of which are actually based off of Express. When you need to download Express you can do it easily with NPM. A framework is a pre-written library of code that you can make use of. In this case, express is a library of node.js functionality available to you in shorter, more easy to use, code.

The term boilerplate is synonymous with template, and is often used in the tech world. I am going to provide a boilerplate server.js file and talk about a few of its pieces because I think this can be hard to find.

The Node.js Express boilerplate includes:

server.js code

before this will work we need to write our index.html file, which our server is set up to serve, I will do this in the client side section. We also have to initialize this directory as a node project with the following command.

$ npm init

you will be asked a series of questions, the defaults are fine, just keep pressing enter. Then finally we have to make sure express is part of the project with the following command.

$ npm install express –save

You must run server.js as a daemon, or background process if you are on a remote server (which your app eventually will be) so that it doesn’t quit after you log off, which is what would happen if you simply used the following command.

$ node server.js

In order to do this search for and install “Forever” through npm. To use it, once it is installed, whenever you run your server, instead of typing the following.

$ node server.js

Type this next line instead.

$ forever start server.js”

Lines 1 and 2 import express and then set it to be used with the app variable. app.get on line 25 is how it generally looks to receive and respond to a client request. If on the client side there had been code that sent a get request without a special path beyond the home url, or with one that didn’t match any other get request, this catch all would be triggered (I will talk more about this in the client side section). It’s a catch all statement because the statement “*” is where you put what request path this code will respond to and the star represents any path, so this will respond to any request and belongs at the end of your server file so that it will fire if no other request statements are triggered. Another request header, replacing line 25 might look more like line 16 or like the following:

app.get(‘/alternatepathhere’, (req,res) => {

Line 4 – 7 actually create an HTTP server and set it to listen on port 3000. Alternatively we could have used very similar looking code to set up an HTTPS server as long as we had an HTTPS certificate we previously obtained through an HTTPS certificate provider, which can be done for free through ‘cerbot’, google it.

Lines 9 – 22 will allow you to connect to a mongoDB database if you replace “my-database” with whatever you named your DB when you created it through the mongo client, same with “collection”, and if somewhere before this you include the line below.

var MongoClient = require(‘mongodb’).MongoClient;

Notice that line 19 looks a lot like what you would use in the mongo client to look up data. Well, here you can use that code to send data back to the client that sent a request to the path ‘/url/to/connect/to/database’.

This Mongo connection is assuming that you are running in auth mode and require a username and password to connect. If you are not running in auth mode that you can leave the username and password out.

If we have created an HTTPS server that means that all of our communication from our client side code to our server and vice versa will be encrypted. The other main security issue that we need to consider is the ability to access our Application Programming Interface (API), which is just a way of saying make requests of our server. earlier I talked about how you can set a particular path to a set of server code (again, this will make more sense after reading the client side section because you will get to see how a request is made). So the security issue is that in many cases, we don’t want anyone to just be able to request this code of our server. This is where setting up a secure session between your client side and server side code enters the conversation. This is beyond the scope of this document, however it is something I would like to write more about in other posts. The other main security consideration is that all user inputs have proper validation so that on the client side users can only put what you want them to into your system (I will talk more about this in the client side section).

Section 5: The Web App Client Side

Client Side Language

My running example for client side programming language will be javascript which is by far the most popular client side language, however there is also coffee script, and typescript.

I am not going to go into detail about the syntax of HTML CSS or Javascript. I am going to lay the foundational concepts that will allow me to continue to give a complete picture of a webapplication. For the specifics of syntax there are many amazing resources, I will talk about some of them, but also, if you search you will find more than you know what to do with.

Hyper Text Markup Language – HTML – this is how you label everything for the browser, and for the rest of the code you will be writing. For example, if you call a block of text a paragraph by using the tags that follow then you can refer to it with CSS and javascript.

<p>your content here</p>

Also, if you give it an id or class you will have further options when referring to it like the following example.

<p class=”myParagraphClass” id=”myParagraphID”>your content here</p>

Once you have labeled all your content with HTML, it then exists as something called the Domain Object Model (DOM). This means that programming languages are able to identify and change objects that you have created. Cascading Style Sheets (CSS) allow you to select an object in the DOM, and then change it’s appearance, event to the extent of animating something or responding to screen size.

Anything beyond changing appearance is the realm of javascript, for example, manipulating data and asking other APIs for data through HTTP requests. If at any point you want to change the appearance of your application because of an event or change in data, you can effect both the HTML of your application and the CSS through javascript. Javascript can also detect if the user does something, like click the mouse button on a particular object in your DOM.

I am going to provide an example template that shows what I just talked about in context, but keep in mind that it will change a little in the next section because the front end framework that is part of my running example, Angular, requires things to look slightly different.

If I run my server.js file (which serves my index.html file) the following is what displays in the web browser at ‘localhost:3000’

Client Side Framework

My running example for a client side framework will be Angular which was developed by google, this most popular alternative to this is React which is facebook’s project. Angular has a Command Line Interface which means that angular provides some useful tools available to you through the command line. I will talk more about some of the commands later in this section. You should download and install Angular-CLI using npm

Once angular has been installed, in whatever directory you would like your app to be created you can type the following command.

$ ng new <name of your app>

Keep in mind that things are going to work differently in this section than was described in the previous section because angular has created a template for you and changed the syntax a little, for example you will notice that angular has already created a CSS, HTML and typescript file and they are all linked together for you. All you need to do is put your HTML in the HTML file, the CSS in the CSS file and you will be able to reference it from the typescript file (.ts). Don’t be afraid of typescript, it is only javascript with added functionality.

Application Programming Interface (API) is the face that a program exposes to the world to let other programs interact with it. Let’s say that you wanted to build an app that needs information from Facebook, Facebook’s API is what you would have to become familiar with so that you could interact with it. HTTP verbs are how you will be talking to an API, they are Get (gets information) Post (adds new information) Put (changes existing information), and Delete (deletes existing information).

One of the best parts of using a framework like angular is that we will now have access to its methods like HTTP, in order to communicate with our server’s API. Angular is also really great for displaying data, it has special notation that you can add right into your html to allow data in your typescript to show up. For a full walkthrough of angular I recommend the book, “ng-book 5”, it is 5 as of writing this, but that number is changing rapidly. That’s not something to be scared of because as long as you are using angular rather than angularjs you can expect most things to be compatible between versions, angularjs was the first version, and it is significantly different than angular 2 up through the current version.

Section 6: Web Hosting

My running example for a web host is going to be digitalocean.com. They start as low as $5 per month for service and they have a really fun feature to play around with called “one click apps” which means that you can create a server with a fully installed WordPress, MEAN, or a variety of other instances.

So, You will sign up and create a “droplet”, you can choose the $5 option to begin with. Digital Ocean has a lot of great help documentation, so I don’t want to go into too much detail. But basically once you get a server up and running, you’ll want to use the IP address they give you to log into the server. To do this open up a CLI on your machine and type the following command, replacing “IPAddress” with the new server’s IP address.

ssh root@IPAddress

you will be asked to change your password from the original one they gave you, which should have come in an email. Once you do that you can change your directory to where you would like to clone your app that you have waiting for you at github.com, this will make more sense after you read the “Version Hosting” section. After this all you need to do is run your server file and then, rather than visiting your app at “localhost:3000” you can access it over the web by entering the IP address of your server into your browser. Lastly, in the Domain Name Registry Section I will talk about replacing this IP number with what you are used to seeing, a web address like youraddress.com that will direct to your web app.

Section 7: Version Hosting

In the most basic sense, versioning software allows you to work on your web app, push it to your version host, my running example will be github.com, and then pull it from github.com onto your remote server. This is useful because it is much better to work on your app from your laptop for example but have it reside on your hosts server. This workflow allows you to use whatever development environment you would like and then when the work is done, with the push of a button you can sync the version of your app on the server with the version you were just working on. The only versioning software I will talk about is git and the only service I will talk about that uses git is GitHub.com. This is because they are far and away the most popular.

A repository is a particular location that versioning software has stored your web app and cloning a repository is the act of creating a repository of your project, on your remote server for example. Github.com has very good instructions for how to start and clone a repository. However, it can be difficult to find good instructions for the basics of keeping the repositories synched with one another so I will talk about that a little. An IDE can automate a lot of this workflow. But, without one the following is how to synch a local repository (the one on your laptop) to a remote repository (the one on GitHub).

If the web app project you have on your local machinge is not currently a github repository, then follow github’s instructions to make it one. This will include creating a repository at github.com and then, on your local machine adding a remote origin that points to that repository. Then from your local machine, type the following within the repository.

$ git add –A (this tells git to pay attention to all the files in the current directory)

$ git commit -m “<message>” (this tells git to “stage” all the changes you have made regarding the files you specified in your last command, and the message is not optional, if you don’t include one then a CLI text editor will open so that you can enter one)

$ git push remote origin (this tells git to update your remote repository that you pointed it to in the previous steps)

Now that the repository on GitHub has your latest work, you would then go to your host server, go to the directory where this project has already been cloned, or do so if this hasn’t been done, and get the latest version from GitHub using the command that follows:

$ git pull remote origin

As for the basics, that is all I will be talking about here. Without further knowledge of the workings of git it is important that you don’t directly change the files on your host server, only work on the files on your local machine and then use the previous processes to update your web app on your host server. This is because, git will not successfully pull to your host server if it detects additional changes in the files on your host that were not already on your Github repository.

Section 8: Domain Name Registry

My example web app uses hostgator.com, a popular alternative is godaddy.com. First you need to make an account, then search for available domain names, if you find one you like, you can buy it. Then under the Domain tab, go to your new domain and under “Name Servers” replace what is there with three new entries “ns1.digitalocean.com”, “ns2.digitalocean.com”, and “ns3.digitalocean.com”. Now, back at digital ocean we need to go to the networking tab, locate your droplet and then add an “A” record. Enter the domain name you just bought from Hostgator and tell digital ocean what IP address you want it to point to. Wait a while, generally 24 hours or so for all of this to take effect and then you should be able to visit your server by entering your domain name in a browser.

Leave a Reply

Your email address will not be published. Required fields are marked *