I recently joined a DevOps challenge aiming to provide interested participants with 20+(?) projects to build and develop their DevOps skills. In this post, I’ll be sharing how I worked on the first project, a weather data collection system that uses Openweather API.
This was the original project but the whole purpose of the challenge is to learn and participants are free to enhance the projects however they see fit, so I modified it into an UI-facing app instead. I also deployed the app into the AWS ecosystem and mostly used scripts to batch the entire process after knowing the steps I wanted to carry out.
Concepts Covered
Python app development (Streamlit)
Infrastructure as Code (Python SDK, AWS CLI commands)
Cloud Storage (AWS S3)
Containerisation (Docker)
Container App Deployment (AWS ECS with Fargate)
CI/CD (GitHub Actions, AWS)
App Features
Fetches real-time weather data for any city of choice (input via UI)
Displays weather conditions like temperature (°F), humidity, etc
Automatically saves weather data as JSON files in AWS S3 with timestamps for historical tracking
1) Building and Running the App Locally
The full script was already provided, so I focused on modifying it to work as a Streamlit app; my full codes can be found on GitHub.
I modified the
requirements.txtto includestreamlitand one other dependency (pyarrow) that kept causing an issue.Run python apps/scripts in virtual environments to ensure installed dependencies are isolated, so:
# Create an environment e.g named .venv
virtualenv -p python3 .venv
#activate
source .venv/bin/activate
Run
pip install -r requirements.txtStart app with
streamlit run <file_name>
2) Running as a Container
Since I also wanted to deploy the app, I tried running it as a container, so I created a Dockerfile. You can also follow the instruction guide here.
FROM python:3.9-slim
COPY . /app
WORKDIR /app
RUN pip install --no-cache-dir -r requirements.txt
EXPOSE 8501
HEALTHCHECK CMD curl --fail http://localhost:8501/_stcore/health
ENTRYPOINT ["streamlit", "run", "dashboard.py", "--server.port=8501", "--server.address=0.0.0.0"]
Build the image and run (—-env-file needs to be specified so that your app knows where to look to load the environment variables).
# Build image
docker build -t streamlit_app .
# Run container app
docker run --env-file .env -p 8501:8501 streamlit_app
3) Deploying to AWS
Since I wanted to deploy to AWS as a container app, some easy options were likely Elastic Beanstalk and App Runner. However, I wanted to practise working with Amazon ECS so I went with it instead.
To deploy the app to ECS, you need to:
set up a repository in Amazon ECR and then push the image to the repository;
setup a task definition in ECS and create a cluster, then create a service in the cluster using the task definition.
There are some configurations that need to be done to use ECS but I had done something similar before (albeit unsuccessful), so these permissions and roles were already setup. However, I still added them in the scripts that I used.
Run the scripts using sh <script_name> in the correct order (all scripts used can be found here).
Everything worked successfully and I could access my running app via <ip-address>:8501 (that was the port I mapped in the task definition).
(Not sure why the app GIF is not showing up but it’s also in the GitHub repo).
Note: using the IP address as-is leads to an issue when updating the service (forcing a new deployment to use the latest image for the task definition if your Docker image has been rebuilt).
The task is replaced hence the IP address changes, so the recommended approach is to configure and use a Load Balancer. This will route traffic to the right running task and its DNS name remains static, so there will be no need to worry about the changing IP addresses of the tasks
Enhancements
Setup an Application Load Balancer to properly route traffic from the running task without needing to look for its IP address.
Create a GHA workflow to run the scripts on push to repo (i.e rebuild the image, push, and update the service)
I also wanted to add a feature to retrieve five most recent data added to S3 for a specified city, but my focus was less on the app development and more on successfully working through the full end-to-end process i.e development to deployment.
Overall, I’m quite happy with my additions and I got to learn more about deploying containerised apps to ECS.