1 - Semaphore NetOps
Deploy and manage Ansible Semaphore on your server. The Netos NetOps project lays the foundation for you to install and manage NetBox, and other network tools and labs.
- Getting Started
- Deploy Netos Semaphore
- Deploy Semaphore & Netos Projects
- Sync Semaphore Repository Updates from Netos
- Deploying NGINX & Certificates
- Manage & Operate Semaphore
Getting Started
Introduction to the Netos NetOps Project
Introduction
Our goal with our Sempahore projects is to deliver a framework that makes it simple and fast to deploy and manage open-source enterprise networking applications and tools.
- Quickly spin up an enterprise grade application stack (in minutes).
- Deal with all the critical yet time consuming platform engineering at the click of a button.
- Deal with the boring but important stuff like backups, restores, and maintenance.
- Fast track to the interesting stuff like playing with the tools and dev/test.
Over time we will publish more projects with associated playbooks. Currently available is:
- Netos NetOps - A simple wrapper project for managing the underlying Semaphore instance.
- Netos NetBox - Contains all the playbooks to deploy and operate NetBox.
Ansible Semaphore
Semaphore UI is a simple and easy to use alternative to Ansible AWX. Netos have used it for over a year without issues to run thousands of playbooks and manage all of our infrastructure.
It is convenient and easy to use - user friendly web interface for executing Ansible playbooks, Terraform, OpenTofu, Pulumi code and Bash scripts. It is designed to make your automation tasks easier and more enjoyable.
Playbooks
At a click of a button you can perform complex installations, like installing NetBox, and a lot more.
Views
Views group together different playbooks, for example, below you can the tasks that deploy NetBox:
Semaphore Projects
Finally, we group everything together into different projects.
Key Semaphore Principles
Introduction
The following diagram illustrates the purpose of each menu item in Semaphore, in the context of the Netos deployment.
Environments
Environments pass variables into Ansible (or other scripts such as Python or Bash).
For example, in this environment file we pass through variables from the Semaphore Environment, but also set default variables where applicable in the /vars/
files in the Ansible project.
CERT_CONTENT: "{{ lookup('env', 'CERT_CONTENT') | default('must-be-set-in-semaphore-variable') }}"
CERT_DIR: "{{ lookup('env', 'CERT_DIR') | default('/netos/certs/netbox') }}"
DOMAIN: "{{ lookup('env', 'DOMAIN') | default('netos.dev') }}"
NETBOX_DB_NAME: "{{ lookup('env', 'NETBOX_DB_NAME') | default('netbox') }}"
Semaphore also supports encrypted secrets as variables. We decided to use this feature throughout rather than Ansible Vault because it ensures that all data is in one place, rather than being split across different vaults and environments.
There is a bug in Semaphore where updates to Secrets are not saved. Problem: not possible to edit secret name · Issue #2293 · semaphoreui/semaphore (github.com)
Repositories
Repositories are linked to Templates (e.g. an Ansible playbook) and in general point to a Git repository. In the case of all Netos projects, we instead point to a local file system, and instead have a per-project task to pull the repository from Netos Networks (github.com).
Templates
Everything above is pulled together into a Template.
Scheduling
Templates can then be scheduled for repetitive tasks, such as backup and housekeeping tasks.
Tracking Tasks
You can globally track the status of all tasks, as well as within the logs of each Template.
Users, Accounts & Tokens
Introduction
We have set default tokens and passwords to get a test/lab environment up and running quickly. In a production deployment you should replace all the tokens and passwords. We don't suggest changing any directories.
Default Login Values
The default accounts to login to systems following deployment from Semaphore are as follows. The endpoints (URLs) are set in Semaphore environments, and you will need deploy and configure NGINX.
System | Username | Password | Endpoint Settings |
Semaphore | admin |
admin |
Set in Semaphore environment "Semaphore Global Settings" |
NetBox | admin |
ohp8toef7Jee |
Set in Semaphore environment "NetBox Global Settings" |
Airflow | admin |
admin |
Set in Semaphore environment "Pod Global Settings" |
Kibana | elastic_admin |
ahee0JeebieB |
Set in Semaphore environment "Pod Global Settings" |
Elastic | elastic_admin |
ahee0JeebieB |
Set in Semaphore environment "Pod Global Settings" |
Var Files
There are different Ansible Variable files located in each project, for example:
- https://github.com/netos-networks/netos-netops/tree/main/vars
- https://github.com/netos-networks/netos-netbox/tree/main/vars
Here is an example vars_file
from the netos-netbox
repository. The "lookup" value is taken from.
CERT_CONTENT: "{{ lookup('env', 'CERT_CONTENT') | default('must-be-set-in-semaphore-variable') }}"
CERT_DIR: "{{ lookup('env', 'CERT_DIR') | default('/netos/certs/netbox') }}"
DOMAIN: "{{ lookup('env', 'DOMAIN') | default('netos.dev') }}"
NETBOX_DB_NAME: "{{ lookup('env', 'NETBOX_DB_NAME') | default('netbox') }}"
NETBOX_DB_PASSWORD: "{{ lookup('env', 'NETBOX_DB_PASSWORD') | default('VxW6EnnKRrkxCzcnDnWT8Fz9q') }}"
NETBOX_DB_USER: "{{ lookup('env', 'NETBOX_DB_USER') | default('netbox') }}"
NETBOX_HOSTNAME: "{{ lookup('env', 'NETBOX_HOSTNAME') | default('netbox') }}"
NETBOX_INSTALL_DIR: "{{ lookup('env', 'NETBOX_INSTALL_DIR') | default('/opt/netbox') }}"
NETBOX_REPO: "{{ lookup('env', 'NETBOX_REPO') | default('https://github.com/netbox-community/netbox.git') }}"
NETBOX_TOKEN: "{{ lookup('env', 'NETBOX_TOKEN') | default('d4c5b00f7053317be2ce8993dd74caa14ca53ca8') }}"
ORG_NAME: "{{ lookup('env', 'ORG_NAME') | default('Netos Networks') }}"
PLUGIN_ADD_TAG: "{{ lookup('env', 'PLUGIN_ADD_TAG') | default('') }}"
POSTGRES_USER_PASSWORD: "{{ lookup('env', 'POSTGRES_USER_PASSWORD') | default('3SqtYWH8iy0Y1alOIj2I') }}"
PRIVATE_CERT_CONTENT: "{{ lookup('env', 'PRIVATE_CERT_CONTENT') | default('must-be-set-in-semaphore-variable') }}"
SECRET_KEY: "{{ lookup('env', 'SECRET_KEY') | default('ahz3ool4teiNgo7moh6fiehiuTh6zei5achae2eeshae9vaiYe') }}"
SUPER_USER_EMAIL: "{{ lookup('env', 'SUPER_USER_EMAIL') | default('netbox@netos.dev') }}"
SUPER_USER_PASSWORD: "{{ lookup('env', 'SUPER_USER_PASSWORD') | default('ohp8toef7Jee') }}"
SUPER_USER_USERNAME: "{{ lookup('env', 'SUPER_USER_USERNAME') | default('admin') }}"
AIRFLOW_API_USER_USERNAME: "{{ lookup('env', 'AIRFLOW_API_USER_USERNAME') | default('airflow_api') }}"
AIRFLOW_API_USER_PASSWORD: "{{ lookup('env', 'AIRFLOW_API_USER_PASSWORD') | default('a17baa2b642565b1d7be4d6d52a7fc23a2c6c41a') }}"
AIRFLOW_API_TOKEN: "{{ lookup('env', 'AIRFLOW_API_TOKEN') | default('d6d52a7fc23a2c6c41aa17baa2b642565b1d7be4') }}"
Semaphore Variables
Variables and Secrets are set in Environments which are passed as variables via the lookup
command above. Check the Key Semaphore Principles guide for more information about Semaphore.
The variable must be configured in Semaphore if it is referenced in a variable file. If it isn't you will get an error when running the playbook.
In a production system, double check all accounts after setting them. I.e. ensure you can manually authenticate to databases, APIs, applications, etc.
Deploy Netos Semaphore
Deploy Semaphore & Netos Projects
Introduction
This guide details how to install Ansible Semaphore and automatically import all of the projects referenced in Netos NetOps. For example, to be able to Install NetBox, run through this process first.
1. Size Your Server
To run Semaphore and NetBox we'd suggest a server with a minimum of 4 x vCPU and 6GB of RAM, with SSD or NVMe disk.
In the future Netos will publish more comprehensive guidance on server sizing. Many variables need to be taken into account. For example, a static NetBox instance with 100k devices doesn't require many system resources (they are just rows in database tables after all), whereas a NetBox server with 1k devices that is getting hammered by constant API hits, many plugins, and many CPU intensive background jobs is a different story.
2. Install Ubuntu 24.04 LTS
The Netos Semaphore solution has been developed and tested only on Ubuntu 24.04 LTS (Long Term Support), which you can download from here. The ISO image will look like: ubuntu-24.04-live-server-amd64.iso
.
Netos Semaphore will not work on earlier versions of Ubuntu.
3. Install Semaphore
Run Installation Script
On a fresh install of Ubuntu 24.04 LTS with internet access, paste this script.
sudo wget https://raw.githubusercontent.com/netos-networks/netos-netops/refs/heads/main/easy-deploy.sh && sudo chmod +x easy-deploy.sh && sudo ./easy-deploy.sh
Alternatively, if you want to clone the repo yourself, follow these exact steps:
sudo mkdir /netos
cd /netos
sudo git clone https://github.com/netos-networks/netos-netops
cd netos-netops
sudo chmod +x deploy.sh
clear
sudo ./deploy.sh
The /netos/
directory is central to many features, and in many cases (such as hard coded Semaphore variables) it's not easy to change, unless we start running sed
on SQL dumps. Hence, please stick to the directory structure!
Installation Process
The deploy.sh
script will:
- Install dependencies on Ubuntu using
apt
- Install Ansible via
pip
into avenv
- Run the
semaphore-install
Ansible playbook which will:- Install MySQL and provision the database
- Install Semaphore
- Import the Netos Semaphore projects from a SQL dump (
semaphore-netos-netops.sql
)
- Present you with a URL to login to Semaphore to continue with the setup, e.g. https://10.1.1.1:3000
Re: step 4.3, we decided to import the Semaphore projects using a SQL backup rather than using the Semaphore UI Restore API. The Restore API doesn't support importing secret variables, which would have therefore required a significant amount of work for new users to just get a base instance up and running. You can of course (and should) change all the default passwords in a production build. Over time we'll transition to the Restore API.
Netos maintain a master/clean instance of the Semaphore project and create the semaphore-netos-netops.sql
file as updates are made. If you want to add playbooks or tasks to the project, please create an issue in Netos NetOps GitHub.
4. Installation Screenshots
The following screenshots show the installation process when deploy.sh
is run on the Ubuntu 24.04LTS server.
Note the URL you can connect to and to use TCP port 3000. Note that in the NGINX configuration guide, you can present this correctly using a FQDN on TCP443.
5. Connect to Semaphore
Once installed you should connect to a page like this, where you can login with the following default credentials. Check the Users, Accounts, and Tokens guide for more information about credentials and variables.
- Username:
admin
- Password:
admin
6. Delete Unwanted Projects & Tasks
You can delete unwanted projects, views, and tasks from the default Netos installation. When you pull repository updates, as outlined here, it will not change the Semaphore configuration. For example, maybe you want to delete these, if deployed:
Sync Semaphore Repository Updates from Netos
Introduction
Each Semaphore project links to an associated GitHub repository, for example:
- NetOps = https://github.com/netos-networks/netos-netops
- Netos NetBox = https://github.com/netos-networks/netos-netbox
When you run playbooks from within the projects, e.g. "Deploy NetBox" it is running the playbook locally, i.e. on the local filesystem. This is by design to give you control over pulling updates from GitHub, and to ensure stability in your environment. Over time more playbooks will be added and fixes made.
From within each SYSTEM view of each Semaphore project is a task named something like "Git Pull Project_Name Repo". Run this to pull the latest updates for that Semaphore project.
The local directory structure on the server looks like this:
- NetOps =
/netos/netos-netops
- NetBox =
/netos/netos-netbox
- Netos Pod =
/netos/netos-pod
- Netos Airflow =
/netos/netos-airflow
Don't edit or make changes to the repository directories as they will be force replaced each time you pull the repo.
The repository directories are excluded from the Semaphore backup job in the NetOps project.
Deploying NGINX & Certificates
Introduction
We have packaged pre-configured NGINX configurations for all applications that can be deployed through Semaphore, e.g. Semaphore, NetBox, Airflow, Elastic, and so on. Within each Netos defined project in Semaphore, there will be a "SYSTEM" tab which is where you deploy NGINX and certificate settings.
Configuration
NGINX
You can find the nginx.conf
and {{ app_name }}.conf
files in the nginx-*
roles in the Ansible playbooks in the Netos GitHub repositories.
We suggest using our templates because each application has different requirements, for example, Semaphore requires WSS configurations for HTMLX, and NetBox requires /static/
to be correctly mounted. We have also configured backend keep-alives from NGINX to NetBox.
Custom vs. Self-Signed Certificates
In a lab environment a self-signed cert is ok (it'll give you a security warning when connecting via the browser). In each Semaphore project you'll see tasks like these which will provision the NGINX configuration and certificates.
Setup the Semaphore Environment
The variables are as follows. At a minimum we suggest setting the domain and hostname in ALL projects before deploying any applications, i.e. this example is for Netos NetOps, but you should do the same in Netos NetBox, etc.
DOMAIN
= e.g.netos.io
(don't put dots or asterisk at the start)SEMAPHORE_HOSTNAME
= e.g.server01-semaphore
CERT_DIR
= This is where the certs will be stored. We suggest leaving everything in/netos/
as we have defined.ORG_NAME
= e.g. Netos Networks LtdCERT_CONTENT
= the key chain for your private certPRIVATE_CERT_CONTENT
= the private cert (see example below)
The hostname must be unique for each project, e.g. server01-ansible.netos.dev
and server-01-netbox.netos.dev
are permitted, however server-01.netos.dev
for both Semaphore and NetBox would result in a validation check failure when running the playbook. This is because NGINX cannot have multiples sites with the same name bound to the same port, e.g. TCP/443.
If you want to add your own cert (or wildcard cert), e.g. signed for server-01-ansible.yourcompany.com
, then you would paste something like this into the PRIVATE_CERT_CONTENT
, and the longer key chain one in CERT_CONTENT
.
-----BEGIN CERTIFICATE-----
MIIDqzCCApOgAwIBAgIUXdRrdhnsq0u8C7vVGjNcFk7zR2wwDQYJKoZIhvcNAQEL
BQAwgZ0xCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEPMA0GA1UEBwwGTG9zIEFu
Z2VsZXMxHDAaBgNVBAoME0V4YW1wbGUgQ29ycG9yYXRpb24xITAfBgNVBAsMGERl
dmVsb3BtZW50IERlcGFydG1lbnQgMTEUMBIGA1UEAwwLRXhhbXBsZSBDQTEeMBwG
CSqGSIb3DQEJARYPYWRtaW5AZXhhbXBsZS5jb20wHhcNMjQwOTIyMTAwMDAwWhcN
MjUwOTIxMTAwMDAwWjCBnTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMQ8wDQYD
VQQHDAZMb3MgQW5nZWxlczEcMBoGA1UECgwTRXhhbXBsZSBDb3Jwb3JhdGlvbjEh
MB8GA1UECwwYRWR1Y2F0aW9uIFRlY2hub2xvZ3kgRGVwdDEYMBYGA1UEAwwPRXhh
bXBsZSBUZXN0IENsaWVudDEeMBwGCSqGSIb3DQEJARYQdXNlckBleGFtcGxlLmNv
bTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABDEiOb9bbErfJbPm0pIR0nA2zJKV
X2gTr1tHcBRCzMt8hjd7Fi5I66qjGVOlkg0HZykOZ98ByU9k6FvZDxG6RCejUzBR
MB0GA1UdDgQWBBRifOqHPJ1T/pPq3c5QnU9i77NmnTAfBgNVHSMEGDAWgBSePRPc
Kt+UjPnlhqFXONvT4GlGRDANBgkqhkiG9w0BAQsFAAOCAgEAjMsdNWRtV7zxW9M7
OZ/ZwLzC+FUd87F2ghy0tFzbfP+j9PKbftmF7HJTrFv9uv2d2L6y7JnTRy0+STNY
1hcmPlbbjiFiRt9bgj5xdtxXnZG2D6UG8QZVwv03AeX4YsUmo5RQxOjWxIwjlVpU
ddp9XpXg2aQsVc/ir9vPjJgCV3FXLw5Q57QCRtncFRksQ/XcdQw53bBBnMiUPzqD
hOZSPTe/yYwa1YeeguxFos5QQwtfrK7Tnllng31dSu+TljRt4LrUgTgP44egKARx
/+XX0p7xsIgFb6pAKk/JUgA4SYerH2UjRyb1D0NcBOoRFPOpe1LtI+fyRYDOiZ/b
d26w/TG0YB5bz9D56PA12KEXoB0CX5tr/1zJlfSITLlRr+HpEgfX3mVfxJ6Dd5yP
9tyBYV7Z7TYPbh4D5wGROVbVmsxVc0mub5TeFg2yLUIBG/fOpU4gfJ7p7A36G5Wd
fEp51bLxCU+yFkJlLQXeJqEd9RdTHKewgHCNrzTL54W2gF+l7udRYVDDWkTvN+p5
q5+Z23kmH2sz8cpIpRUhGRZ1lTtyokDUIHKZl+cx5IckvA4pkLzZm8eFODG1+7vH
2kPUuOme4mB8o2tLnpq2u2NcHBrcNPg4Tl5zFMvZnsXpT6n8UPMRZMtZ8i8S6GgD
NPZxatAb0KiO2SHfCquCpCqmdYMe01TIbWU=
-----END CERTIFICATE-----
Deploy the Certificates
In our case we have copied a wildcard full-chain cert and private key for *.core.netos.io
into the environments in Semaphore.
After the playbook has deployed NGINX and the certs, you can change the domain in the URL and refresh the page.
Note that we configured DNS in advance of deploy NGINX and the certs. If you are running a local lab, you could set the hosts file on your workstation to point to the respective services. For example, like this:
10.1.1.1 uk-lab-nb05-semaphore.core.netos.io
10.1.1.1 uk-lab-nb05-netbox.core.netos.io
Troubleshooting
If things went wrong, you can always connect on the IP https://x.x.x.x:3000
URL to get into Semaphore, and delete the certs and NGINX site configuration file if required.
As long as you can connect to Semaphore, you can keep running the playbook to re-apply the settings, which will replace the certs and NGINX config files.
Then restart the NGINX process once the above files are deleted: sudo systemctl restart nginx
.
Manage & Operate Semaphore
Semaphore Backup Guide
Introduction
The Semaphore backup process provides regular secure backups. In summary the playbook:
- Manages retention based on your preferences, e.g. deletes backups older than 14 days old
- Backs up and encrypts the
/netos/
directory, which contains many important things, such as:- Certificates (for NGINX)
- Past NetBox plugin ZIP file downloads
- Working directories for other applications and tools
- Backups (which are excluded from the backup)
- Backs up the
/etc/nginx/
directory, which contains all the site configurations for different services deployed by Semaphore, such as Semaphore itself, NetBox, Airflow, etc. - Backs up and encrypts the
semaphore
MySQL database - Optionally SFTP's the backup files to a secure remote SFTP server
The Semaphore process and MySQL database is NOT stopped during the backup.
Semaphore uses MySQL and not PostgreSQL to ensure isolation from other applications running on the server. For example, NetBox and Airflow both use PostgreSQL, and if/when those databases are restarted, we don't want to impact the management wrapper, i.e. Semaphore.
Cron Scheduling
The cron scheduler in Semaphore can be configured to backup the database at regular intervals, for example, at 02:30 every day. You can check the Dashboard page in the menu, or the task history to check the outcomes.
Note that there is a bug in Ansible Semaphore UI that causes the same task to run many times. The solution is to toggle the "Show cron format' button and use UNIX formatting like here.
Backup File Rotation
You can set the retention period for backup files stored in /netos/backups/semaphore
in the Environment / variables.
Ensure you use the exact values of days
or weeks
, i.e. no capitals.
Remote SFTP
To enable remote SFTP, change the No
value in the SFTP_ENABLED
variable to Yes
, and set the SFTP_HOST/USER/PASS
values accordingly.
An example of the encrypted backup folder contents is as follows:
Restore Semaphore
Manual Restore
We haven't included a playbook in Semaphore to restore itself, because it's a bit like sawing the branch off that you're hanging from. To restore from your Semaphore backup, you can follow this process.
- Start with a fresh install following the Deploy Semaphore Guide, except CTRL-C and exit the installation.
- Search and replace the
semaphore-netos-netops.sql
file in the/netos/netos-netops/roles/semaphore-install/files
with your SQL backup and restart the install usingdeploy.sh
- Restore the tar.gz file to
/netos/
- Restore the tar.gz file to
/etc/nginx
- Restart the server and connect to Semaphore
Decrypting Backup Files
If you want to manually decrypt the enc
files on your local workstation, use the following commands.
openssl enc -aes-256-cbc -d -in BACKUP_FILE.enc -out OUTPUT.sql.gz/.tar.gz
You will need the password set in the Semaphore Backup Settings Semaphore environment variable ENCRYPTION_KEY
.