The Nervous System Architecture: Privacy with Agentic AI Without Sacrificing Intelligence

At first, integrating agentic AI into my workflow seemed inevitable, yet something held me back from beginning my learning journey. I desired the intelligence these agents provided, but as I delved into their autonomous nature and features, my paranoia about the fate of my sensitive data grew. I experimented with hosting local models, which offered peace of mind regarding information security, but, without sufficient hardware resources, I found that they lacked the efficiency required for certain tasks. So, when I turned to the cloud for more horsepower, I asked myself a straightforward question: how do I keep my sensitive data private?

This article is as much a record of my self-study on securing Agentic AI infrastructure as it is a guide on how to start to building one yourself. Over time, I developed a repeatable baseline and I've decided to call it The Nervous System Architecture. It's a framework I built for myself—one hybrid infrastructure where "local" and "cloud" stop being mutually exclusive and start working together under my control.

Agentic AI is powerful, but without rigorous guardrails, it invites critical data leaks. A local model is not a panacea. Even behind a firewall, a model with persistent memory could still be exploited by a single malformed prompt into releasing proprietary secrets.

As I began experimenting with agentic AI, it became clear over time that true data leakage often occurs not at the model, but through its tools. Integrating external APIs or third-party search services creates gaps in your defense. These are the essential "crumbs" of data—IP addresses, credentials, API keys, and URLs—that can potentially exit your secure perimeter.

The core idea I'm presenting in this piece is simple: privacy isn't about where the model sits—it's about controlling where the data goes. Map your data flows, separate intelligence from execution, and you build the foundation for a true privacy architecture.

This leads to the nervous system model—a framework in which the "brain" can be any capable API, while the nervous system—the data, the tools, the environment—stays entirely under your control. In this model, efficiency no longer requires exposure.

To make this work, I needed to understand where the line actually is.

The Fallacy of the Local Boundary

It is a common mistake to view the firewall as the finish line. When we grant an agent the power to act, we must be mindful of where its knowledge travels and who—or what—may be listening and collecting it at every stop, regardless of where the model is hosted.

photo credit: Ted Balmer, Unsplash

Privacy, therefore, is not a static state of being "local"—it is the cumulative result of the entire data flow.

This realization shifts the equation: an agent is only as useful as its security posture is comprehensive. For those operating environments within strict governance and compliance frameworks, this complexity is a significant barrier. While agentic AI promises to automate the most repetitive, time-consuming workflows for organizations, its adoption is lagging. The hesitation isn't due to a lack of capability, but because of the sheer complexity these agents add to a modern information security program.

This leads us to a paradox of deployment. The very tools that make agentic AI a "force multiplier"—web scraping, browser automation, and real-time API integrations—are the same tools that can puncture the local perimeter. The appeal is obvious: by leveraging these tools, a business can avoid the massive overhead of hiring a team to fine-tune a proprietary in-house model just to achieve basic agency.

But here is the friction: the model is only one part of the stack. When an agent is capable enough to interact with the cloud to get the job done, it is capable of carrying your data with it.

This leaves us with the central question of the modern AI architect: Is it possible to maintain absolute privacy without sacrificing the function that makes agentic AI valuable in the first place?

Mapping the Flow: From Blind Deployment to Structured Infrastructure

The answer is yes, but it requires a fundamental paradigm shift.

True privacy is not achieved by simply locking the door to the room where the model lives; it is achieved by controlling every pipe, wire, and signal that enters and exits that room.

When you shift your focus from the location of the model to the pathway of the data, the binary choice between "local" and "cloud" disappears.

The first step in this transition is moving from blind deployment to a mapped infrastructure. You cannot secure a flow you cannot see. This requires a ruthless inventory of your AI stack—not just the model, but the retrieval mechanisms, the tool-calling interfaces, and the persistent storage buffers.

It is during this mapping process that the "local" illusion truly shatters. You discover that while the model your agent is leveraging is on-premise, its tools may not be. Every time your local agent calls a third-party search API or a cloud-based scraper to gather information, it could be leaving a trail of sensitive data with every tool call. These are the essential "crumbs" of data—queries and URLs—that can potentially exit your secure perimeter. These crumbs may seem trivial, but in the aggregate, they form a legible map of your proprietary research, your internal interests, and your strategic direction.

To stop the leak without killing the function, we must first audit the flow and choose our partners wisely. Start by mapping your data path visually—draw a diagram or network topography that notes every single point along the route, from prompt generation to retrieval, tool execution, and output. Think critically about each hop; don't just mail it in.

You can read more about self-auditing your AI environments in this article.

If you choose a cloud model, pair that map with a ruthless review of data sovereignty. You need to know whether that provider is training on your prompts or retaining your data. I recently discovered that even API services like OpenRouter, which pride themselves on minimal log collection for their services, will still route through backend providers like Anthropic or Google that can be training away on your prompts without you really knowing it. Look for zero-data-retention support and backend settings—tools like OpenRouter offer privacy toggles. Read the privacy statements. Choose providers that actually value your privacy and demonstrate it by offering solutions.

Before designing the boundary, it helps to name who's on the other side. Three threat profiles matter here:

The provider — the API operator who could log or train on your prompts. This is what I'm solving for with zero-data-retention policies and backend opt-outs.

The network — traffic between your agent and the API could be intercepted. TLS handles this at baseline, but for high-assurance work, a private network or trusted egress point is better.

The compromised endpoint — a malicious webpage, poisoned tool response, or supply-chain attack that exfiltrates data through the agent's own tools. This one cuts both ways: it doesn't matter if the brain is local or cloud if the nerves are compromised.

This article focuses on the first one — the provider — because that's where the nervous system architecture delivers the most utility for a jump start into implementing agentic AI. The other two are real problems, but it should be noted that they need additional layers on top of this foundation.

The Nervous System Architecture

Instead of trying to force every single component to be local—which often results in a crippled, low-capability agent—you architect a hybrid system. You utilize a high-capability neural model through an API connection for the "brain," providing the reasoning and intelligence. But you keep the "nervous system"—the environment where the agent lives, the sensitive data it accesses, and the tools it uses entirely private and local.

In this model, the "brain" processes the logic, but it never touches the raw, unshielded perimeter. The nervous system handles the interaction with the world, ensuring that the data crumbs never leave your sight. You no longer have to choose between the power of the cloud and the privacy of the local; you simply define where the boundary actually lies.

Here is a simple example of how this architecture works in practice.

My Hermes agent runs on a Fedora VM that lives in my homelab—or could eventually live in the cloud (Azure, AWS). This VM is the nervous system's central node: it holds the files, manages the sessions, and controls the environment. No one else has visibility into what happens here. For web searching, I self-host Firecrawl in a Docker container right alongside it, and it performs wonderfully. The agent itself is linked to OpenRouter via API, which routes my requests to the various Qwen and Deepseek models. I chose OpenRouter because I like that they have published a clear policy: any data collection is opt-in. They do not share, sell, or license underlying prompt data to any third party, and they make it transparent which of their model providers support zero-data retention (so you know which ones you can safely send prompts to). The "brain" gets the intelligence it needs through an API connection, but every tool call, every search query, every file it reads or writes stays within the system I control. The brain thinks, the nerves touch, and the data stays home.

Conclusion

The nervous system architecture didn't start as a thesis. It started as a practical question: how do I build this for myself without giving up control?

Today my Hermes agent runs on a Fedora VM in my lab, queries the web through a Firecrawl container I own, and talks to models through a provider whose privacy policy I've actually read and agreed with. The intelligence comes from the cloud. The data never leaves my perimeter. I didn't compromise capability for privacy, I just stopped treating them as a trade-off. Putting it together myself made all the difference in how I understand the technology.

That's the whole point of this article. I know there are others out there considering these kind of tools for themselves and their businesses. I started this journey with (what I believe to be) a reasonable amount of paranoia. I'm hoping that sharing this architecture might help bridge a mental gap.

If you're considering building with agents, you might be facing the same question. You don't have to choose between intelligence and privacy. You just have to decide where the line really is—and then protect it.

Discover

The first step in securing generative AI workflows is to discover all instances of AI agents across the environment. Without a clear picture of where these tools are being used, businesses cannot effectively manage risk.

Discovery starts with building a comprehensive inventory of systems, applications, and binaries where AI models, libraries, or agent frameworks may be deployed. Vulnerability scanning should extend beyond just network-level scans. Effective discovery for generative AI requires file system–level inspection to capture details about individual binaries, dependencies, and libraries.

Specialized tools that generate a Software Bill of Materials (SBOM)—such as the open-source tool Syft—can play a critical role in this process. An SBOM provides visibility into every component that makes up an application, making it easier to identify outdated or vulnerable dependencies. While SBOM tools are excellent for software composition analysis, discovery should also leverage more traditional endpoint detection and response (EDR) and vulnerability management tools. These solutions perform deep scans of endpoints and cloud VMs, and while they may not natively generate SBOMs, many support exporting their generated results in open standards like CycloneDX and SPDX. These outputs can then be fed back into SBOM tools for further enrichment and analysis.

Technical scanning, however, only tells part of the story. Many employees use generative AI in ways that may not be captured by endpoint scans or SBOM analysis—for example, through their web browser or approved software that interacts with an LLM model via API calls. For some businesses, especially those with heavy SaaS (software-as-a-service) adoption, this may actually be the most common way generative AI is used day to day. These activities often remain invisible, which is where we get the term “shadow AI”.

To fill this gap, organizations should supplement technical discovery with employee surveys or interviews. By asking staff how they use AI in their daily work, security teams can uncover shadow AI practices and gain insight into business-driven use cases that technical scans alone would miss.

Assess

Once discovery is complete, the next step is to assess the AI security posture (AI-SPM) of your environment.

This means evaluating the overall state of an organization’s security configuration, readiness, and risk exposure, and then establishing a systematic way to continuously monitor, assess, and improve that state. Think of it as a “health check” or ongoing “fitness plan” for your environment, including generative AI workflows.

Assessing posture requires a comprehensive approach to maintaining the integrity, confidentiality, and availability (CIA) of generative artificial intelligence and machine learning systems. It involves building a strategy for continuous monitoring and improvement of the security posture of the AI models, services, and workflows found in the infrastructure. AI-SPM focuses on spotting and fixing weak spots, misconfigurations, and risks associated with AI usage while also ensuring and/or maintaining compliance with privacy, security, and ethical standards for the organization’s industry.

Several frameworks and tools can help structure this process:

• NIST AI Risk Management Framework (AI RMF) – provides a foundation for identifying and managing AI-specific risks.

• OWASP AI Maturity Assessment (AIMA) – a useful resource for measuring the maturity of AI security practices.

• SBOM scanning and remediation reviews – ensures that vulnerabilities found in AI-related components are addressed, tracked, and prioritized. Automation is certainly encouraged here wherever possible to make assessments repeatable and auditable.

Assessment should not stop at static analysis. Grey-box penetration testing provides an additional layer of assurance by simulating real-world attack scenarios. A well-scoped pentest can help uncover vulnerabilities that are specific to the environment, validate whether the remediation efforts were effective, and probe the resilience of AI systems against bad actors trying to manipulate the system or execute model-specific exploits.

Ultimately, assessment acts as the bridge between discovery and control. This step transforms raw visibility into actionable security intelligence.

Control

After discovery and assessment, the next step is to apply security controls to generative AI workflows.

Controls should incorporate prompt-level security—intercepting user inputs and analyzing them for prompt injection or jailbreak attempts.

An AI prompt is the input given (by the user) to a large language model via a generative AI platform.

Prompt injection is an attempt to manipulate an LLM model to make it do something harmful. This is the number one method bad actors use to manipulate generative AI according to the OWASP Top Ten. An attacker might use social engineering techniques to guide generative AI, giving instructions to override its original commands. This could lead to unintended actions, such as leaking sensitive internal information or executing unauthorized operations. By exploiting vulnerabilities in the prompt handling, attackers can potentially bypass security measures altogether.

Tools like Cloudflare’s WAF can act a proxy, inspecting layer 7 traffic and enforce gen-AI specific rulesets, blocking malicious prompts before they reach in-house or third-party models. This same visibility also helps address shadow AI, where unapproved AI tools may bypass traditional defenses.

Equally important are privacy safeguards. Generative AI must be configured to protect sensitive data, such as PII or PHI, and ensure compliance with the framework adopted during the assessment phase.

Finally, organizations should enforce outbound controls to monitor and restrict what data leaves the environment, reducing the risk of unintentional disclosure through AI services.

In short, controls provide the guardrails that keep AI secure, compliant, and aligned with organizational risk objectives.

Reporting

The final step in securing generative AI workflows is risk management and compliance reporting.

Organizations should create dashboards that visualize and prioritize AI-related risks, making it easier to track vulnerabilities, remediation efforts, and overall security posture.

For industries with regular compliance requirements, these reports are especially valuable, simplifying audit preparation and demonstrating adherence to regulatory standards. A structured reporting process ensures that stakeholders have clear visibility into both the current state of AI security and ongoing mitigation efforts.

I will be demonstrating the CIS benchmark role with a newly installed Ubuntu 22.04 host, implementing the level 1 and some level 2 controls. If you’d like to dig deeper, I’ve included links to the project’s GitHub repository and official documentation.

This guide assumes you are familiar with Ansible and the CIS benchmarks, and that Ansible is already installed and working on your machine.

Why automate compliance?

So, why might you be looking to automate compliance in the first place?

As more organizations shift workloads to the cloud—especially in SaaS—security becomes a critical focus. Compliance requirements are increasingly strict, particularly in regulated industries like healthcare, finance, and government. To meet these demands, many teams are choosing to integrate preconfigured cloud system images in these environments since they come with software packages and/or security-focused configurations that provide a jump start to defining the base configuration for system resources.

In AWS, for example, you’ll find CIS Hardened Images—AMIs (Amazon Machine Images) that are purpose-built for secure, compliant environments. For example, you can find the CIS Hardened Image Level 1 for Ubuntu in the AWS marketplace. These images are incredibly convenient, but they come at a cost: higher hourly operating fees and annual license costs. For large-scale or cost-sensitive environments, this pricing model can quickly become a burden. This not a viable solution for hybrid environments where AMIs are simply not available to deploy for the on-premise machines.

System administrators who have tried it know that hardening a system by hand is very tedious work and is very time-consuming. Doing it manually across multiple systems? That is a tremendous headache.

Enter Ansible.

Ansible gives you the power to define and enforce your security baselines from a single control node, pushing configuration changes across hundreds—or thousands—of machines. This is why Ansible is now considered a necessary skill for DevSecOps engineers. Automation not only speeds up deployment but it makes compliance repeatable, auditable, and scalable.

Automating security hardening is a practical approach to compliance. Unlike using a pre-configured image, it allows for complete customization and greater control right from the beginning.

The Ansible Lockdown project takes this a step further by offering compliance-as-code.

What stood out to me is how accessible and transparent it is. As an engineer who likes to test and experiment in my home lab first, I appreciate being able to pull the roles directly from GitHub and get a clear overview from the documentation. It’s well-organized, actively maintained, and offers solid coverage for a range of operating systems.

This tool is particularly attractive if you're seeking an open-source alternative to prebuilt cloud images or costly compliance solutions and are already using Ansible. It's flexible for hybrid environments and, being based on Ansible, integrates smoothly with existing automation workflows.

This post is my first hands-on experience with the role, and I’m exploring it from the perspective of someone who’s considering it for use on future projects.

Let’s get started.

Setup

The Ansible Lockdown role works best when applied to a freshly installed system with a base configuration. Running it on a production server, or any system that already has applications in place, can and likely will cause services to break. This is because the role enforces strict security configurations that can interfere with existing setups.

Before running the benchmark, make sure your server is set up with the required disk partitions, as specified in the CIS Benchmark controls. For level 2, you’ll want the following mount points:

• /var

• /var/tmp

• /var/log

• /var/log/audit

• /home

These partitions are required so the benchmark can enforce mount options like nodev, nosuid, and noexec, which play a role in preventing privilege escalation and unauthorized code execution.

Before running the playbook, make sure that root access is configured with a password on the target host and that SSH key-based authentication from your Ansible control node is working properly.

You might also consider setting a GRUB password to harden the bootloader. While the Lockdown role can configure this for you, it’s worth noting that this control (along with a few others) is disabled by default. That’s because improperly setting a GRUB password can lock you out of the system if it’s misconfigured.

Installation

You can install the Ansible Lockdown role directly from the project’s Git repository using the ansible-galaxy command. This tool is used to manage Ansible roles and collections, making it easy to pull in reusable automation code from external sources.

The Ansible Lockdown project is structured as a single repository that contains multiple roles for different operating systems. When you install a role using ansible-galaxy, it organizes the role’s tasks, handlers, and defaults into a structure that Ansible can manage.

Here’s how you can install the Ubuntu 22 role from GitHub:

ansible-galaxy install git+https://github.com/ansible-lockdown/UBUNTU22-CIS.git

This will place the role in the current user's ~/.ansible/roles/ directory.

Auditing

To evaluate the effectiveness of the role, I’ve used Wazuh to monitor my Ubuntu 22.04 instance. Wazuh is a free and open-source security platform used for threat prevention, detection, and response across various environments, including on-premises, virtualized, containerized, and cloud-based workloads. Wazuh includes its own CIS compliance assessment module, which runs checks based on the CIS benchmark.

Before applying the Lockdown role, the system scored a 48% compliance rating out of the box. I’ll be using Wazuh again after applying the playbook to see how much that score improves.

One of the most useful features of the Lockdown role is the ability to run it in audit-only mode. This allows you to check your system against the CIS benchmark without making any changes, which is great for testing.

You can configure audit mode either by passing variables directly on the command line with -e like this:

ansible-playbook -i /etc/ansible/hosts -e '{ "setup_audit":true,"run_audit":true, "audit_only":true, "grub_user_pass":  }' ./site.yml -K

Or, you can directly modify the role’s configuration file.

~/.ansible/roles/UBUNTU22-CIS/defaults/main.yml

Set the following variables to true:

setup_audit: true
run_audit: true
audit_only: true

Also, set the get_audit_binary_method to "download" so that the required GOSS binary (used for audit checks) can be fetched automatically from GitHub:

get_audit_binary_method: "download"

Once these settings are in place, you’ll be ready to perform your first audit without making any system modifications.

One of the many configurable options in main.yml is whether to set a GRUB password.

However, I ran into an issue when I left the grub_user_pass variable unspecified. The playbook failed to run, which may be a bug, though it’s possible I misunderstood the intended behavior. I plan to report this on the project’s GitHub repository.

Fortunately, there’s a simple workaround:

• If no GRUB password is configured on your system, you can pass an empty value for grub_user_pass when running the playbook:

    -e '{ "setup_audit":true,"run_audit":true, "audit_only":true, "grub_user_pass":  }'
  

• If there is GRUB password set, pass it in like so:

    -e '{ "setup_audit":true,"run_audit":true, "audit_only":true, "grub_user_pass":your_secure_password  }'
  

This avoids the failure and allows the role to handle GRUB settings correctly based on your input.

Alternatively, you can run the GOSS audit from the host using the run_audit.sh script. This is done after running the playbook once to set up the audit files. By default, these files are placed in the host’s /opt directory, but you can change this in main.yml.

Here is an example of an audit summary after the playbook completes:

Remediation

To move from audit mode to actual remediation, you can run the playbook using a similar command as before—just make sure to set audit_only to false.

ansible-playbook -i /etc/ansible/hosts -e '{ "run_audit":true, "audit_only":false, "grub_user_pass":  }' ./site.yml -K

Alternatively, you can update the main.yml file directly.

As the playbook runs, there may be errors that come up such as this error regarding the auditd service:

My experience was that I encountered some errors whenever a control could not be enforced due to a conflicting system state or a missing dependency.

In these cases the options are to:

• Fix the issue, and then re-run the playbook. On subsequent runs, Ansible will skip controls that have already been applied.

• Temporarily disable problematic controls in main.yml, and circle back to resolve them manually later.

In my case, after just a couple of remediation runs (each taking about 3–4 minutes), the system's compliance score improved significantly. Wazuh now reports 75% compliance, up from the initial 48%.

We can now see the benefit of automation in the hardening process—what could have taken days to implement manually is handled in minutes with repeatable, testable code.

Audit again

To get a more comprehensive view of the system’s compliance, I will scan this instance using Lynis, another automated hardening and auditing tool. Lynis is a well-established, open-source project licensed under the GPL and first released back in 2007. It’s widely respected and actively maintained.

What makes Lynis especially useful is that it evaluates your system against multiple security standards—including CIS, NIST, NSA, and OpenSCAP—providing a broader, standards-based perspective beyond what a single benchmark offers.

In a real-world environment, it’s not enough to say a system is “probably secure”—you need proof. Demonstrating compliance is make-or-break for passing audits, especially in regulated sectors. That’s why tools like Wazuh and Lynis are part of my workflow here. They give you third-party visibility into how well the hardening actually aligns with the standards. Running both gives you a fuller picture of where your host stands and helps catch anything that might’ve slipped through. And Lynis is lightweight and fast to get going, which makes it a no-brainer.

Installing Lynis and running a quick scan is straightforward and lightweight. Here’s how to get started.

git clone https://github.com/CISOfy/lynis

cd ./lynis

./lynis audit system -Q

Lynis also reports similar results, highlighting the hardening index. This helps confirm that several controls are indeed implemented on this system.

Once the Ansible Lockdown playbook completes, it generates a .json report in the host’s /opt directory. This file includes the results of every control that was tested, giving you a clear view of what passed, what didn’t, and what still needs attention.

This is very helpful for fine-tuning your configuration, validating changes, and identifying anything you might want to address manually.

After just a few more minutes of manual tuning, I saw another bump in compliance—Wazuh is now reporting 81% against the CIS benchmark. That’s a significant jump from the original state, and it highlights just how effective this kind of automation and iterative testing can be in creating a new baseline.

Conclusion

The Ansible Lockdown project by Mindpoint Group is a solid way to jump-start the hardening process in Linux environments. It’s flexible, accessible, and fits naturally into an infrastructure-as-code approach.

But let’s be clear—compliance isn’t a one-size-fits-all checklist. It needs to be considered from the start of the system’s lifecycle and shaped around the organization’s unique business rules and regulatory obligations. Automation should support that strategy, not define it.

With a bit of planning and customization, this role can help enforce real-world compliance standards in a way that’s repeatable, scalable, and cost-effective.

In that piece, I demonstrated how we can use the pandas library in python to traverse a deeply nested .json object, extract data from specific dictionary keys, and restructure it as a flattened pandas dataframe. From there, we can use sqlalchemy to open a connection to a MySQL database and insert the data into a table.

Now, I want to go further by invoking a Lambda function that can run this Python code, allowing the data collection, transformation, and loading to be completed within seconds using the power of serverless. To get this working, the Lambda function will need to connect to my VPC so that it can talk to an RDS instance in a private subnet. It will also need internet access so that it can gather the source files from an S3 bucket in my account.

My goal in this post will be to walk through how to use Terraform to configure a VPC with public and private subnets, and create a Lambda function that is ready to go— with its deployment package, VPC access, and connectivity to private resources. I will confirm the Lambda function’s connectivity to the RDS instance with a simple python script that utilizes pymysql.

Prerequisites

This guide assumes that you already have some working knowledge of AWS, the Lambda service, and Terraform. If you want to follow along, you'll need to have an AWS account set up and have Terraform and the AWS CLI installed on your computer.

We will need to set up these prerequisites from the AWS console before writing any .HCL code:

• Set up an AWS user account for Terraform.

• Create an IAM role for the Terraform account (with a trust policy that allows the Terraform user account to assume it).

• Set up the appropriate permission policies for the Terraform IAM role.

Provider and authentication configuration

If you’re not familiar with using Terraform, this is quick overview on how Terraform connects to the cloud and creates our infrastructure. Terraform uses what’s known as a “provider” to connect to orchestrate the environment. Providers are a logical abstraction of an upstream API. They are responsible for understanding API interactions and exposing resources.

In this case, we need to create a providers.tf file or write a provider block in the code. For my demonstration, I am going to create a providers.tf file.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region  = "us-east-2"
  shared_credentials_files = ["path/to/my/.aws/credentials"]
  profile = "terraformuser"

  assume_role {
    role_arn = "arn:aws:iam::###:role/demo_role"
    session_name = "demo_role_session"
    }
}

In this file, I have specified shared_credentials_files to pass the credentials that Terraform will use for authentication as well as specified the IAM role Terraform will use.

After this is done, run the command terraform init to initialize Terraform in your working directory.

VPC configuration

By default, Lambda functions are launched in the Lambda managed VPC which has internet access. In order to set up Lambda functions that can access our resources in a VPC, we need to assign the Lambda function to our VPC. Attaching the function to the VPC means that it will be assigned a private IP address and the Lambda function will not have internet access. This is true even if the function is in a public subnet (a subnet with a route to an internet gateway). If the Lambda resource needs internet access from the VPC, it must use a NAT Gateway or a VPC endpoint for AWS managed services. For this demonstration I will use a NAT Gateway.

Let’s start by writing out the VPC configuration. Create a file in your working directory called main.tf. The following creates a VPC, an internet gateway, a NAT gateway, an elastic IP, a public subnet, and two private subnets (I will also be deploying an RDS instance).

#VPC
resource "aws_vpc" "demo_vpc" {
  cidr_block           = "10.20.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "demo vpc"
  }
}

#subnets
resource "aws_subnet" "demo_public_subnet" {
  vpc_id                  = aws_vpc.demo_vpc.id
  map_public_ip_on_launch = true
  cidr_block              = "10.20.1.0/24"
  availability_zone       = "us-east-2a"

  tags = {
    Name = "public-subnet"
  }
}

resource "aws_subnet" "demo_private_subnet" {
  vpc_id            = aws_vpc.demo_vpc.id
  cidr_block        = "10.20.2.0/24"
  availability_zone = "us-east-2a"

  tags = {
    Name = "private-subnet"
  }
}

resource "aws_subnet" "demo_private_subnet2" {
  vpc_id            = aws_vpc.demo_vpc.id
  cidr_block        = "10.20.3.0/24"
  availability_zone = "us-east-2b"

  tags = {
    Name = "private-subnet2"
  }
}

#igw

resource "aws_internet_gateway" "demo_igw" {
  vpc_id = aws_vpc.demo_vpc.id

  tags = {
    Name = "demo-IGW"
  }

}

#EIP
resource "aws_eip" "elastic_ip" {
  depends_on = [aws_internet_gateway.demo_igw]
  domain     = "vpc"
}

#NAT gw
resource "aws_nat_gateway" "demo_nat_gateway" {
  allocation_id = aws_eip.elastic_ip.id
  subnet_id     = aws_subnet.demo_public_subnet.id

  tags = {
    Name = "demo_nat_gateway"
  }


  # To ensure proper ordering, it is recommended to add an explicit dependency
  # on the Internet Gateway for the VPC.
  depends_on = [aws_internet_gateway.demo_igw]
}

After this, we will need to create the route tables and route table associations to complete the VPC configuration. I won't include my code for this step because it's straightforward if you follow the documentation. I also assume that most readers of an article about Terraform configuration have done this before in some way. Still, it's good to provide a complete overview.

This is a good time to test if our Terraform is configured properly and the authentication we set up is working.

Running the terraform plan will command creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure.

When terraform plan runs successfully, it will normally create a terraform.tfstate file and output a preview of the changes it will make in the console. This is what it will look like if there are no changes to make:

AWS Lambda Layers

A Lambda function requires you to specify the runtime as well as provide any dependencies that are needed for the code to run. These dependencies are normally delivered with the function code in a .zip file referred to as the package deployment.

As an alternative to this standard approach, you can create a Lambda layer. A Lambda layer is a.zip file archive that contains supplementary code or data. These layers usually contain library dependencies, a custom runtime, or configuration files.

However, just zipping the …/site-packages directory of your Python project and uploading it to Lambda may not let the code run because of compatibility issues. If you are working with Python functions, like I am, it's useful to know that Python packages with compiled code are not always compatible with Lambda runtimes. I found this to be the case with pandas. If you are getting a “Unable to import module" error after invoking a Lambda function and after verifying that the module in question is included in the Lambda deployment .zip, it may be a compatibility issue.

In order to work with these kind packages in Lambda, it is recommended to install the manylinux versions of these packages (.whl) in your local environment and create the deployment package with those. This article provides a step-by-step guide on how to do this: https://repost.aws/knowledge-center/lambda-python-package-compatible

To create my deployment, I first checked that pip version 19.3.0 or higher was running on my machine. Then I used this bash command to install the dependencies to the target folder /python in the current directory.

pip install \        
    --platform manylinux2014_x86_64 \
    --target=python \
    --implementation cp \
    --python-version 3.11 \
    --only-binary=:all: --upgrade \
    pandas pymysql sqlalchemy

I am not including the function code in the layer so I am not concerned with it for the moment. Once all the necessary packages are installed, run the command zip -r deployment.zip python to create the deployment package.

Go to the AWS Console > Lambda > Layer > Create Layer.

Upload the file from your local filesystem to AWS Lambda and create the layer.

Once this is done, copy the arn of the Lambda layer, we will use it later in our main.tf file.

Note: you can create an aws_lambda_layer_version with Terraform. This is a good option especially if your layer’s .zip package is small. For this to work, you must add a depends_on line in the Lambda function resource block or you may get an error when Terraform is creating the function.

AWS has a 50 MB file size limit for uploading the package deployment file via the AWS CLI. Since my package zip is larger, I chose to use the AWS console and prepare the layer in advance, as shown above. However, since Lambda functions can have up to 5 layers, another valid workaround would be to create multiple, smaller deployment packages.

Creating the function’s IAM Role and security group

To ensure the Lambda function has the correct access to private resources, we need to assign it an IAM role and a security group.

The first thing to do is to define the assume role policy. We can do this in Terraform with a data source.

data "aws_iam_policy_document" "assume_role_policy" {
  statement {
    sid    = "assumerole"
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]

  }
}

We create an IAM role in Terraform with an aws_iam_role resource. aws_iam_role resources require an assume_role_policy which we just created with the data source. We reference this policy document.

resource "aws_iam_role" "iam_for_demofunction" {
  name               = "iam_for_demofunction"
  assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json

}

Next, we can attach the AWS managed policies needed to Lambda to access resources in a VPC using aws_iam_role_policy_attachment.

resource "aws_iam_role_policy_attachment" "demofunction_vpcexecution_policy_attach" {
  role       = aws_iam_role.iam_for_demofunction.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}

resource "aws_iam_role_policy_attachment" "demofunction_basicexecution_policy" {
  role       = aws_iam_role.iam_for_demofunction.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

Because my Lambda function needs to be able to access the project’s S3 bucket and connect to an RDS instance, I opted to include these additional policies separately and attach them in the same way.

data "aws_iam_policy_document" "demofunction_inline_policy_document" {

  statement {
    sid    = "allRDS"
    effect = "Allow"

    actions   = ["rds:*"]
    resources = ["*"]
  }

  statement {
    sid    = "allactionsS3projectbucket"
    effect = "Allow"

    actions = ["s3:*"]
    resources = ["arn:aws:s3:::bucket_name",
      "arn:aws:s3:::bucket_name/*"
    ]
  }

}

resource "aws_iam_policy" "demofunction_inline_role_policies" {
  name        = "demofunction_inline_role_policies"
  description = "Adding additional inline role policies for demofunction"
  policy      = data.aws_iam_policy_document.demofunction_inline_policy_document.json
}

resource "aws_iam_role_policy_attachment" "demofunction_inline_role_policies" {
  role       = aws_iam_role.iam_for_demofunction.name
  policy_arn = aws_iam_policy.demofunction_inline_role_policies.arn
}

Next, we can create a security group for the Lambda function. I will also be creating a security group for my RDS instance (which is the private resource my Lambda function connects to).

resource "aws_security_group" "demo_lambdasg" {
  name        = "demo_lambdasg"
  description = "security group for lambda"
  vpc_id      = aws_vpc.demo_vpc.id
}

resource "aws_security_group" "demo_dbsg" {
  name        = "demo_dbsg"
  description = "security group for database"
  vpc_id      = aws_vpc.demo_vpc.id

}

Finally, define the egress and ingress rules that allow the Lambda function to connect to the database.

resource "aws_vpc_security_group_ingress_rule" "allowtraffictodbfromlambda" {
  referenced_security_group_id = aws_security_group.demo_lambdasg.id
  from_port                    = 3306
  to_port                      = 3306
  ip_protocol                  = "tcp"
  security_group_id            = aws_security_group.demo_dbsg.id
}

resource "aws_vpc_security_group_egress_rule" "allowoutboundtrafficfromlambda" {
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = -1
  security_group_id = aws_security_group.demo_lambdasg.id
}

Creating the function

With the functions layer, IAM role, and security group created, we can write the resource block for the function itself. Here, we have a Lambda deployed with a python 3.11 runtime and a 15 second timeout. You can create a zip file of the function’s code and enter the path using the filename argument. This could also be s3_bucket or image_uri. You must select one of these.

resource "aws_lambda_function" "demofunction" {
  filename      = "/path/to/function/index.zip"
  function_name = "demofunction"
  layers        = ["arn:aws:lambda:us-east-2:######:layer:layer_name:#"]
  role          = aws_iam_role.iam_for_demofunction.arn
  handler       = "index.lambda_handler"

  runtime = "python3.11"
  timeout = 15

  vpc_config {
    subnet_ids         = [aws_subnet.demo_private_subnet.id]
    security_group_ids = [aws_security_group.demo_lambdasg.id]
  }

   environment {
       variables = {
         DBUSER     = ###
         DBPASSWORD = ###
         DBHOST     = your-instance-name.your-region.rds.amazonaws.com
       }
     }
}

For layers, paste the arn string for the corresponding layer that was collected in the previous section.

Reference the subnet_id for the private subnet and the security group to complete the VPC configuration.

I’ve included environment variables so that Lambda can login to the RDS instance in my VPC. For better security, you could use the AWS SSM parameter store to securely access the DBUSER and DBPASSWORD variable since these values are sensitive. In Terraform, this can be referenced with a data source. If you also used Terraform to create the RDS instance, you can use the address attribute of the database resource to reference its DNS name.

Alternatively, this looks like:

environment {
       variables = {
         DBUSER     = data.aws_ssm_parameter.DBUSER.value
         DBPASSWORD = data.aws_ssm_parameter.DBPASSWORD.value
         DBHOST     = aws_db_instance.demo_database.address
       }
     }

Save the file.

Apply

We can run terraform plan again to see what changes Terraform will make to our state. This time it shows all of the planned changes dictated by the main.tf file.

Now, run terraform apply -auto-approve and wait for Terraform to provision the resources.

I will run a quick test to ensure that this is working as intended.

My Lambda function will run this simple script to connect to the RDS instance and create a table named “users”.

import pymysql
import os

def lambda_handler(event, context):

    connection = pymysql.connect(
        host=os.getenv("DBHOST"),         
        user=os.getenv("DBUSER"),     
        password=os.getenv("DBPASSWORD"), 
        database="test", 
        charset="utf8mb4",        
    )

    try:
        # Open a cursor to perform database operations
        with connection.cursor() as cursor:
            # Write an SQL statement to create a table if it doesn't exist
            create_table_query = """
            CREATE TABLE IF NOT EXISTS users (
                id INT AUTO_INCREMENT PRIMARY KEY,
                name VARCHAR(100) NOT NULL,
                email VARCHAR(100),
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
            """
            # Execute the SQL statement
            cursor.execute(create_table_query)

        # Commit the transaction to apply changes
        connection.commit()
        print("Table 'users' created successfully!")
        return 200

    except pymysql.MySQLError as error:
        print("Error while connecting to MySQL or executing the query:", error)
        return 500

    finally:
        connection.close()

Calling the function from the AWS console returns a 200 "OK," indicating that the connection was successful and the table was created.

The end.

To sum up, the Lambda function triggers logic that queries the jawa.gg API for its current GPU listings and collects the API response as a JSON file, which is then uploaded to an S3 bucket. I chose jawa.gg for this project because it intersects with a couple of my hobbies. jawa.gg is a marketplace for second-hand PC components. I enjoy playing games in my spare time, and I also like learning about and collecting PC hardware. Over the years, I've built several gaming rigs and servers.

This project was undertaken solely for educational purposes.

The problem

The lambda function is running fine but I’ve decided it time to learn about how I could use this data, after all, what’s the point of collecting this kind of data without the ability analyze it? This is what the raw response looks like:

Yuck.

In this post, I will show how I wrote python code that will upload this data to an SQL database where it can be queried and more easily read.

I am using Jupyter Notebook to work with the data. Jupyter Notebook is an interactive web-based tool designed for data science and programming. It offers a versatile environment where a user can write and execute python code in cells. This approach really helps with exploring of the data, and is essentially allowing me to write my code in very small stages while being able to refer to previous cell output.

Understanding the data structure

Let’s get started. In my notebook, I load the file contents and print it so that I can analyze it:

Here is a snippet of the file contents (which is now in variable datafile ).

[[{'blocked_at': '', 'buyer_protection_policy': None, 'category': {'id': 6, 'name': 'GPUs'}, 'condition': 'used', 'created_at': '2024-10-17T00:44:00.193Z', 'description': 'Card has been kept in near perfect condition, with no scratches or signs of wear and tear.\nBox is included, along with the anti-static bag, PCIE pin cover and heavy duty memory foam for protection. Power cables not included.\n\nPurchased from EVGA in March 2022, recently upgraded in May 2024.\n\nRecommended for up to 1440p gaming, where most games achieved around 120fps on max settings for me. Easily overclockable with MSI Afterburner or EVGA software.\n\nAny questions at all, feel free to message me here or at null\n\nThanks!', 'expired_at': '', 'featured_at': '2024-10-17T00:46:29.640Z', 'height': 9.5, 'id': 46535, 'images': {'ids': ['production/listings/nxyjg6c3lrausbvjejcx', 'production/listings/ankxxvb2xrrsjrw4071q', 'production/listings/rnqo4zpslnqpi9osc2nt', 'production/listings/xgehov2xcqosfrz1x01r', 'production/listings/yhbvlhftz4zt058q3zjz', 'production/listings/jmeqfhhy2uzgloh2h0kg', 'production/listings/kxlg1l2jzuubruetc7rz', 'production/listings/glo3rnzp78liniqhmofo'], 'source': 'cloudinary'}, 'is_insured': True, 'is_on_sale': False, 'is_private_listing': False, 'is_published': True, 'is_sold_out': False, 'labels': [], 'last_featured_at': '', 'last_published_at': '2024-10-17T00:44:00.189Z', 'last_sold_at': '', 'length': 15.5, 'listing_code': '', 'listing_expires_at': '2024-11-16T00:00:00.000Z', 'minimum_offer_amount': None, 'name': 'EVGA GeForce RTX 3070 Ti FTW3 ULTRA GAMING, 8GB GDDR6X, ARGB LED, Metal Backplate', 'number_sold': 0, 'original_price': 45000, 'price': 45000, 'price_last_changed_at': '2024-10-17T00:44:00.193Z', 'published_at': '2024-10-17T00:44:00.189Z', 'quantity': 1, 'quantity_available': 1, 'return_policy': None, 'shipping_jawa_manual_price': 0, 'shipping_option': 'jawa_fulfilled', 'shipping_seller_manual_price': 0, 'sku': '', 'specs': {'Chipset': 'Nvidia', 'Brand': 'EVGA', 'Series': 'GeForce RTX 3070 Ti', 'Memory': '8GB'}, 'status': 'available', 'url': '/product/46535/evga-geforce-rtx-3070-ti-ftw3-ultra-gaming-8gb-gddr6x-argb-led-metal-backplate', 'usage': 'TWO_Y', 'user': {'badges': [], 'displayed_name': 'Impassioned-Jawa#4838', 'facebook_ids': ['facd13fa-48f5-4816-98a2-68382f3cf9df'], 'id': 293620, 'images': {'avatar': 'production/avatars/e0d2yifarjwoggcd4x70', 'source': 'cloudinary'}, 'is_business': False, 'is_guest': False, 'is_on_vacation': False, 'is_verified': False, 'store_name': '', 'tag_line': '', 'username': 'Impassioned-Jawa#4838', 'website': ''}, 'user_id': 293620, 'video_url': '', 'watch_count': 0, 'weight': 2, 'width': 4.5},

If we use type , we can further confirm that the very top level of this data is one big list. The first index, datafile[0], is also list. However, datafile[0][0] is a dictionary. So the first realization here is that the data that I am interested in extracting is two levels down. Each dictionary represents one product listing.

Here is another look at the file’s structure from Jupyter Notebook. Here, datafile = root.

Here are the dictionary keys:

datafile[0][0].keys()

So, I need to extract the data from these keys and upload it to the database. The dictionary keys will become the database columns, and each row will represent one GPU listing.

Pandas library

This is where I learned about the pandas utility! I want to shout out YouTuber Panda's Data Diaries. While researching, I found his video and realized that Pandas would be very useful here. Pandas is a python library that can flatten .JSON and turn it into a Pandas “data frame”. The data frame is a relational representation of the data structure. This is very similar to how data shows up in an SQL database. What’s more, Pandas can connect to an SQL database and push the data frame to the database. Perfect! This library is exactly what I’m looking for.

Using the json_normalize method I am able to print out the structure in columns.

#datafile[0] is a list of 20 dictionaries which are individual products
import pandas as pd
df = pd.json_normalize(datafile[0])

I noticed that some columns are still being obscured by an ellipses.

Adding the options pd.set_option('display.max_columns', None) and pd.set_option('display.max_colwidth', None) fix this issue. Now I can see important columns, i.e. the “price” column.

I can use Pandas to connect to my SQL database and directly copy the data to a table, or even create a new table if it does not exist yet.

Transforming the data

Before doing that, I need to transform the data. I don't need certain columns from the data frame, so I will exclude those. Secondly, I need to perform a numerical operation on the numbers in the "price" column so that they appear as float values or the actual price in USD.

I can select a single column by using display and putting the name of the column as a string in-between a set of brackets.

df = pd.json_normalize(datafile[0])
display(df['condition'])

Here is the data frame again with only the columns that I want to keep (some do not appear on screen):

To clean this up further, I later define a list called “selected_columns”. Then, calling display(df[selected_columns]) can produce the same result.

Now that is done, I need to deal with that issue with the “price” column. The “price” column is a string type and it shows as a factor of 100 above the actual price value. So, for example, where the price may show up as “45000” as a string, the actual price is “450.00” as a float.

Here is how I fixed that:

import pandas as pd
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)

#flatten JSON
df = pd.json_normalize(datafile[0])

# Select specific columns
selected_columns = [
    'condition', 'description', 'id', 'images.ids', 'listing_expires_at',
    'name', 'price', 'published_at', 'return_policy', 'shipping_option',
    'specs.Chipset', 'specs.Brand', 'specs.Series', 'specs.Memory', 'weight'
]

# Convert 'price' to numeric and handle any non-numeric values just in case
df['price'] = pd.to_numeric(df['price'], errors='coerce')

#divide 'price' by 100 so that a float value appears for the price
df['price'] = df['price'] / 100

df['price']

Result!

This completes the data transformation. Now I can work on inserting the data into the SQL database.

Connecting to MariaDB

For my initial testing, I set up a MariaDB server on a raspberry pi in a Docker container. When I'm ready to scale up, this will become an AWS RDS instance in the cloud, but for now, I wanted to experiment in a local environment. The database container uses port 3306 on the host network. I deployed the container using the docker compose template available on the official Docker Hub, which also deploys the Adminer container. Adminer is a tool for managing content in SQL databases and features a user-friendly Apache-based GUI, similar to phpMyAdmin.

I am using standard password authentication and everything is set up and ready to go. The database name is ‘gpulistings’, it’s empty with no tables created.

I am following along with the steps in this article showing how we can use sqlalchemy to connect Pandas to an SQL database. I should mention that this article uses a postgres database but I am using MariaDB (MySQL). And so I needed to ensure that I installed pymysql and changed the connection string in the code to use the correct pymysql connector.

The tutorial suggests to write out the db username and password in the command you will use to initiate the db connection. However, this is a bad security practice and I would absolutely avoid this if I was running this in production. So, I will create python environment variables instead so that I can hide these sensitive credentials.

I’ve created a “.env” file in my Python virtual environment and wrote out the variables in that file.

Then after installing python-dotenv, I can import the necessary modules into my code and I’ll do a quick test to check if the variables are available. This is working:

Now I’ll write the code that will create the db connection and insert the data. sqlalchemy really makes this simple.

from sqlalchemy import create_engine 
from dotenv import load_dotenv
import psycopg2 
import os


load_dotenv()
db_user = os.getenv("DB_USER")
db_pw = os.getenv("DB_PASSWORD")
db_host = os.getenv("DB_HOST")


#create a new data frame from the selected data so that I can use the ".to_sql" method
newdf = pd.DataFrame(df[selected_columns])

load_dotenv()

#define env variables
db_user = os.getenv("DB_USER")
db_pw = os.getenv("DB_PASSWORD")
db_host = os.getenv("DB_HOST")


# establish connection with the database 
engine = create_engine( 
    f"mysql+pymysql://{db_user}:{db_pw}@{db_host}:3306/gpulistings") 

#create a table named products and use 'APPEND’ to add data to the existing table.
newdf.to_sql('products', engine, if_exists='append')

Result!

Putting it all together

This is working brilliantly. However, it's only a small part of our data, just 20 product listings. There are over 300 in this file. To finish this, I will write this code as a couple of functions that will go through each list index in the JSON data, create a data frame from the keys in the dictionaries, and then insert that data into the database.

All together this is what it looks like:

from sqlalchemy import create_engine 
from dotenv import load_dotenv
import json
import pandas as pd
import os

#fetchedarray.json is my testing file
with open("fetchedarray.json", "r") as f:
    datafile = json.load(f)

load_dotenv()

#define env variables
db_user = os.getenv("DB_USER")
db_pw = os.getenv("DB_PASSWORD")
db_host = os.getenv("DB_HOST")

def get_dataframe(index):
    pd.set_option('display.max_columns', None)
    pd.set_option('display.max_colwidth', None)

    #flatten JSON
    df = pd.json_normalize(datafile[index])

    # Select specific columns
    selected_columns = [
        'condition', 'id','listing_expires_at',
        'name', 'price', 'published_at', 'shipping_option',
        'specs.Chipset', 'specs.Brand', 'specs.Series', 'specs.Memory'
    ]

    # Convert 'price' to numeric and handle any non-numeric values
    df['price'] = pd.to_numeric(df['price'], errors='coerce')

    #divide 'price' by 100 so that a float value appears for the price
    df['price'] = df['price'] / 100

    newdf = pd.DataFrame(df[selected_columns])

    return newdf


def insertdb():

    # establish connection with the database 
    engine = create_engine( 
        f"mysql+pymysql://{db_user}:{db_pw}@{db_host}:3306/gpulistings") 

    #I found that each API response .json has one empty list at the end. So, length needs to be " -1"
    #to avoid errors caused by the empty list. it just works :)
    for i in range(len(datafile) - 1):
        get_dataframe(i).to_sql('products', engine, if_exists='append') 

#call main insertdb() function
insertdb()

Result!

Now I have a populated SQL database with the product’s name, price, shipping details, listing date, and more. The picture above is after I changed out the .JSON file and ran it again, resulting in 698 rows.

Conclusion

With this proof of concept completed, I plan to make some tweaks to the code to accommodate AWS Lambda and RDS so that this can be truly automated. Thanks for checking out and check back for me soon!

I embarked on this project to deepen my hands-on experience with AWS Lambda and JavaScript. Despite having no prior background in JavaScript, I decided to dive in headfirst because I learn best through practice. I was eager to develop new skills, especially in automation, which is crucial for a cloud engineer. It's important to thoroughly understand automation concepts, even at a foundational level, to effectively manage cloud-based infrastructures.

AWS Lambda, a serverless compute service, allows you to run code without the need to manage or provision the underlying servers. For this project, my main goal was to scrape product listings from an online marketplace using JavaScript code executed in a Lambda function, and then upload the collected data to an S3 bucket. I chose a publicly available website for this task, and I want to emphasize that the project was undertaken solely for educational purposes.

As a hobbyist gamer with a strong interest in PC hardware and building custom gaming rigs, I selected jawa.gg, a marketplace for second-hand PC components, as the target website. My script scrapes current GPU listings, processes the data, and uploads the raw results in JSON format to an AWS S3 bucket, where it’s ready for further analysis and transformation.

As I worked through the data collection process, I discovered that the raw JSON output from the API fetch is a deeply nested object. This means that the data will need to be transformed before it can be uploaded to an SQL database for querying. I’ll provide a brief preview of the data structure at the end, however, I plan to post on this in-depth in a different post.

The project breaks down into two key parts: writing a JavaScript program to collect the data and setting up an AWS stack that includes Lambda for execution, S3 for storage, IAM for authentication, and CloudWatch for monitoring.

When I began this project, I initially opted for a traditional web scraping approach using Puppeteer. However, I quickly realized that this method wasn't well-suited for jawa.gg, as it's a dynamic website. This challenge prompted me to shift to using the API fetch method, which proved to be far more effective for retrieving the data I needed.

The journey has been both challenging and rewarding, and I’m excited to share the insights I’ve gained!

This article is geared towards those with a beginner to intermediate understanding of JavaScript and a working knowledge of AWS at a practitioner level. With that in mind, let’s dive in!

The Web Scraping Challenge: Using Puppeteer

The Puppeteer library is a Node.js tool that provides a high-level API for controlling headless Chrome or Chromium browsers programmatically. It can also run Chrome or Chromium in full (non-headless) mode. Puppeteer is commonly used for tasks such as web scraping, automated testing, generating PDFs, capturing screenshots, and interacting with web pages in a way that simulates user behavior.

I began by writing code for Puppeteer, but eventually, I moved away from it because it couldn't deliver the results I needed. I can show why at the end.

I kicked off coding in a streamlined dev setup, which was a Node.js installation running on my local Windows 11 machine.

//import the puppeteer module. I have included it in the package.json
import puppeteer from 'puppeteer';

//this is the website URL
const url = 'https://www.jawa.gg/shop/pc-parts-and-components/graphics-cards-G2XFDKAS'

/* the main function launches a new browser instance
extracts GPU product information (title, price, image) 
by interacting with the page’s DOM.
Logs the extracted GPU data to the console (without a headless 
browser for debugging purposes). */

const main = async () => {
    const browser = await puppeteer.launch({ headless: false})
    const page = await browser.newPage()
    await page.goto(url)

    const gpuListing = await page.evaluate((url) => {
        //convert the price string to a number
        const convertPrice = (price) => {
            return parseFloat(price.replace('$', ''))
        }
        //find a specific image URL that I found manually after 
        //inspecting the website. I am selecting the URL for the width of 384 px and return it.
        const convertimgSrc = (srcset) => {
            if (!srcset) {
                return null;
            }
              const urls = srcset.split(' '); // Split on spaces to separate URLs

              return urls.find(url => url.includes('w_384')); // Find the URL with "w_384"


        }
        /* maps over the array gpuPods and for each GPU listing (gpu)
           extact the title pthe listing from the HTML element, the price,and the 
            image URL*/
        const gpuPods = Array.from(document.querySelectorAll('.tw-group.tw-relative'))
        const productData = gpuPods.map(gpu => ({
            title: gpu.querySelector('.tw-paragraph-m-bold.tw-mt-3.tw-line-clamp-1.tw-overflow-ellipsis').innerText,
            price: convertPrice(gpu.querySelector('.tw-paragraph-m-bold.tw-order-2.tw-font-bold.tw-text-jawa-blue-500').innerText),
            imgSrc: convertimgSrc(gpu.querySelector('img').getAttribute('srcset'))

            }));


        return productData

    }, url);

    console.log(gpuListing)

    await browser.close()
}

main();

The program exits cleanly, however, these results are incomplete because it only returns the first page of the available listings.

Sample of the data:

[
  {
    title: 'ZOTAC RTX 2080 Ti TWIN FAN',
    price: 330,
    imgSrc: 'https://res.cloudinary.com/jawa/image/upload/f_auto,ar_5:4,c_fill,w_384,q_auto/production/listings/n92a3ylsspxyzouwsriw'
  },
  {
    title: 'ROG Strix GeForce RTX™ 4070 Ti 12GB GDDR6X OC Edition',
    price: 785,
    imgSrc: 'https://res.cloudinary.com/jawa/image/upload/f_auto,ar_5:4,c_fill,w_384,q_auto/production/listings/d7dqykw5lczm8wld3kxf'
  },
  {
    title: 'Sapphire Pulse RX 6600XT',
    price: 165,
    imgSrc: 'https://res.cloudinary.com/jawa/image/upload/f_auto,ar_5:4,c_fill,w_384,q_auto/production/listings/fu6rlj1c3iu0h6is0pga'
  },

//...

I attempted to progress the program so that the puppeteer agent would click on the next page button like a real person would, log the page, and continue this pattern until it had reached the end of the product listings in a loop. However, I didn’t have any success with this approach. After identifying the HTML element that contained the “next page” button on the site, I assigned it to a variable and wrote a while loop that would check if the element was on the page and then use the .click() method to click on it before running the main function again. But I would either run into an error that said …

throw new Error('Node is either not clickable or not an Element');

… or the browser session would only loop between pages 1 & 2 endlessly. I was sure to include .waitForNavigation() but it made no difference.

So, I investigated the website more closely. I noticed two important clues. For one, I could not manually navigate to a page on the site by simply adding the page number at the end of the URL. It always bounced back to the first page.

That prompted me to take a closer look and understand how the site was designed. I began to analyze the underlying logic for navigating pages so I could try to replicate it in my code in the browser dev tools. I noticed that when the user clicks on the next page, instead of loading a new page, it starts a fetch request.

This discovery helped me to confirm that this webpage is actually being loaded dynamically rather than through using traditional URL-based navigation. When you click a page number or "Next" button, a fetch call is made to the backend API (at the /api endpoint) to retrieve the new set of products. This data is then likely injected into the current page without reloading it. Apparently, this technique is called API pagination. If you want to learn about it more in depth, I have linked a resource here that discusses in much more detail.

So, after I had a better understanding of how the website was displaying the product listings, I adjusted my approach to access the site’s API directly and abandoned the traditional web scraping approach with Puppeteer.

Shifting Gears: Querying the API Directly

I rewrote the program to use fetch() to query the API directly.

//At the end I write a .JSON file with the output  
import * as fs from 'fs';

const apiUrl = 'https://www.jawa.gg/api/search?categoryIds[]';  // the API endpoint
let initialPageNumber = 1  // The starting page number for the request is "1", "0" returns code 400
let response = await fetch(`${apiUrl}=6&page=${initialPageNumber}`);
let dataJson = await response.json();
const now = Date.now();


async function getApiData () {
    //an array that will be updated with the product listings from each page
  let productData = [];

  let state = Object.keys(dataJson.products).length !== 0;
  console.log(state);


  while (state) {

    response = await fetch(`${apiUrl}=6&page=${initialPageNumber}`);
    dataJson = await response.json();
    state = Object.keys(dataJson.products).length !== 0;

    let nextPageNumber = initialPageNumber;
    initialPageNumber++;
    //this console log is for debugging.
    console.log(`The page number is: ${nextPageNumber}`);
    productData.push(dataJson.products);

  }

  if (!state) {
    console.error('No more products');
  }

  return productData;

}



async function cleanData(data) {
//write the output as 'fetchedarray.json'
 fs.writeFile((`${now}_fetchedarray.json`), JSON.stringify(data), (err) => {
    if(err) throw err;
    console.log("Successfully saved JSON!")});

}

const apiData = await getApiData();

cleanData(apiData);

The apiUrl variable holds the API endpoint, and we kick things off by setting the page number to 1—starting at 0 throws a ‘400’ error. We grab the first batch of data using fetch and convert the response into a JSON object. The core logic is inside of the getApiData function. Here, we create an empty productData array to store product listings as we loop through the pages. To keep the loop going, the state variable checks if there are still products available.

As the loop runs, each page is fetched, and we log the page number in the console for some helpful debugging—just to confirm it's pulling every page correctly. The product data is added to the productData array, and once we hit a page with no more products, the loop breaks and console logs “No more products”. Finally, the cleanData function steps in to save everything into a fetchedarray.json file. Date.now() is used to capture the current timestamp in milliseconds (since January 1, 1970). This value is being used in the fs.writeFile function to create a unique filename for saving the JSON data. Any errors during the file writing process will be caught and thrown, and a success message is logged when the data is saved.

Using AWS Lambda for Automation and S3 for Storage

This is where the fun in the cloud begins! The first step is to create the Lambda function that will run our Javascript program.

This function's configuration is basic. It's important to assign the correct permissions with IAM. The function's execution role should allow it to access S3 (AmazonS3FullAccess) and write logs to CloudWatch for monitoring (AWSLambdaBasicExecutionRole).

The next thing I needed to do was adjust the execution time of the Lambda function. The default timeout is 3 seconds but this function runs anywhere from about 7-15 seconds depending on how many listings are being fetched. Trying to run the function without adjusting the default timeout setting will result in this error:

Rewriting the code for Lambda using the AWS JavaScript SDK was not too complicated. I followed the AWS documentation. However, calling the AWS S3 submodule from SDK v3 was not as straightforward as I expected. Many guides I consulted suggested using var AWS = require('aws-sdk');, which calls the entire SDK v2. But since version 3 of the SDK is essentially “modularized” to reduce it’s size, you must call the submodules or you will see a “not found” error. Ultimately, this documentation was helpful in the end:

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/introduction/#modularized-packages

const AWS = require('@aws-sdk/client-s3');
//AWS javascript SDK v3
const s3 = new AWS.S3();
//create a new S3 client


exports.handler = async function (event) {
  const apiUrl = 'https://www.jawa.gg/api/search?categoryIds[]'; 
  let initialPageNumber = 1  
  let response = await fetch(`${apiUrl}=6&page=${initialPageNumber}`);
  let dataJson = await response.json();
  const now = Date.now();


  async function getApiData () {

    let productData = [];

    let state = Object.keys(dataJson.products).length !== 0;
    //console.log(state);


    while (state) {

      response = await fetch(`${apiUrl}=6&page=${initialPageNumber}`);
      dataJson = await response.json();
      state = Object.keys(dataJson.products).length !== 0;


      let nextPageNumber = initialPageNumber;
      //turn in to JSON
      //let dataJson = await response.json();
      initialPageNumber++;
      //console.log(dataJson.products);
      console.log(`The page number is: ${nextPageNumber}`);
      //console.log(state);
      productData.push(dataJson.products);
      //console.log(productData);

    }

    if (!state) {
      console.error('No more products');
    }

    return productData;

  }

  const apiData = await getApiData();

  try {
    var params = {
      Bucket : 'fetcheddata',
      Key: `${now}_fetchedarray.json`,
      Body : JSON.stringify(apiData)
    };

    await s3.putObject (params);
    return { statusCode: 200, body: 'File uploaded successfully' };
  } catch (err) {
    console.error("Error uploading file:", err);
    return { statusCode: 500, body: 'Error uploading file' }
  };



}

The getApiData function logic has not changed. The biggest differences here are adding the Lambda function handler and the try catch block where the PUT operation is initiated. Some highlights:

• This imports the AWS SDK client for S3, which allows interaction with Amazon S3.

• A new instance of the S3 client (s3) is created, which will be used to upload the raw JSON file to my S3 bucket.

• Once the API data is collected (apiData), the function tries to upload it to an S3 bucket.

• The params object defines my S3 bucket, the file key (which includes the current timestamp), and the file body (the JSON string of the fetched data).

• s3.putObject uploads the file to S3.

• If successful, it returns a 200 status with a success message.

• If an error occurs during upload, it logs the error and returns a 500 status with an error message.

Let’s see it in action!

Here is the file, ready and waiting in the S3 bucket!

And, here are the logs in CloudWatch.

Analyzing the Data

Taking a closer look at the kind of data I am getting from the API, this is an example of a single product listing. There are over 250 products listings in this fetch.

{'blocked_at': '', 
'buyer_protection_policy': None, 
'category': {'id': 6, 'name': 'GPUs'}, 
'condition': 'used-good', 
'created_at': '2024-07-24T21:35:14.356Z', 
'description': 'This AMD Radeon HD 5870 1GB GDDR5 Graphics Card features DisplayPort, HDMI, and DVI ports. It has been tested, cleaned, and confirmed to be in good working condition, with potential signs of previous use.', 
'expired_at': '', 
'featured_at': '', 
'height': 0, 
'id': 37723, 
'images': 
    {'ids': ['production/listings/qgry6fkvna5423moqtbx', 'production/listings/rhr6groalzpbispxazse', 'production/listings/cmeqwppvlbsqctc8iywh', 'production/listings/d37tvncxckowxh2myb0p'], 'source': 'cloudinary'}, 
'is_insured': False, 
'is_on_sale': True, 
'is_private_listing': False, 
'is_published': True, 
'is_sold_out': False, 
'labels': [{'id': -1, 'name': 'Free Shipping'}], 
'last_featured_at': '', 
'last_published_at': '2024-07-24T21:35:14.356Z', 
'last_sold_at': '', 
'length': 0, 
'listing_code': '', 
'listing_expires_at': '', 
'minimum_offer_amount': 2039, 
'name': 'AMD Radeon HD 5870 1GB GDDR5 Graphics Card -DisplayPort, HDMI, DVI', 
'number_sold': 0, 
'original_price': 2599, 
'price': 2399, 
'price_last_changed_at': 
'2024-07-24T21:35:14.356Z', 
'published_at': '2024-07-24T21:35:14.356Z', 
'quantity': 2, 
'quantity_available': 2, 
'return_policy': {'restocking_fee_percent': 0, 'return_window': 30}, 
'shipping_jawa_manual_price': 0, 
'shipping_option': 'seller_manual', 
'shipping_seller_manual_price': 0, 
'sku': 'AMD HD 5870 1G GDDR5 (2,3)', 
'specs': {'Memory': '1GB'}, 
'status': 'available', 
'url': '/product/37723/amd-radeon-hd-5870-1gb-gddr5-graphics-card-displayport-hdmi-dvi', 
'user': {'badges': [], 'displayed_name': 'TechCastle', 'facebook_ids': ['fd8bd165-53b5-4c9c-a710-da5b6589ae07'], 'id': 215231, 'images': {'avatar': 'production/avatars/deiaybd5arnm9tzr0yam', 'source': 'cloudinary'}, 
'is_business': False, 'is_guest': False, 
'is_on_vacation': False, 
'is_verified': True, 
'store_name': 'TechCastle', 
'tag_line': '', 
'username': 'TechCastle', 
'website': ''}, 
'user_id': 215231, 
'video_url': '', 
'watch_count': 0, 
'weight': 0, 
'width': 0}

This dataset is deeply nested, so extra steps are needed to properly extract the product listing data and prepare it for querying. Additionally, there are a few quirks in the data that must be addressed. For example, while some of the key-value pairs, like 'tag_line,' may be interesting, they are not particularly useful and can be removed. Furthermore, the 'price' value is not displayed in a readable format.

To address these issues, I plan to use a secondary Lambda function to transform the data, leveraging Python which is better suited for the task. We'll reduce it down to only the necessary fields—'name,' 'description,' and 'price.' The price will be converted to an integer and divided by 100 to represent the amount in USD.

The result is a much more useable (this is not the same product show above):

{'name': 'MSI GTX 1650 D6 AERO ITX OC GeForce Graphics Card 4GB GDDR6', 
'description': 'MSI GTX 1650 D6 AERO ITX OC GeForce Graphics Card 4GB GDDR6.\n\nCard was used in an church office computer for around 2 years (only two days a week), just to get 3 monitor support. It was never used for gaming or anything taxing on the GPU.\n\nPulled from that working system and tested to verify functionality.\n\nCard does not require any external power connection. It draws all needed power from the PCIe slot.\n\nDoes not include original box, packaging, or documentation.', 
'price': 89.99},

Reflections and Key Insights from the Process

Facing issues with the Puppeteer approach at first was disappointing, but I'm glad I could switch to a new method for collecting data that worked much better. I discovered that I really enjoy writing JavaScript, and I plan to keep learning more about it. In the coming days, I will need to think like a data scientist and learn how to further transform the data to make it more useful. Incorporating a database would be the best option for data queries.

Stay tuned for more updates on my progress. Thanks for learning with me! If you found this blog interesting, please check out some of my other posts.

Previously, I wrote an article about installing Vaultwarden in a Docker container to securely store my passwords and secret data.

In part 2, I will discuss how I configured the vault and share my experience using it. If you haven’t already, be sure to check out part 1.

Test drive

This is what the vault looks like:

The UI is very user-friendly. To start testing this, I created a log-in item for my Synology NAS.

Vaultwarden is an implementation of the Bitwarden server API and it is compatible with Bitwarden's upstream clients. An example of one of those clients is Bitwarden's browser extension (available for most browsers). There are also options for mobile, devices, desktop applications, and even CLI.

Once the browser extension is installed it is time to flex that API and connect it to my server. When logging in, be sure to choose the option for "self-hosted".

Enter your server's address and your user credentials.

Now you can see your vault in the browser extension and you are ready to use it as your password manager. If you never wanted to log in to the server again you wouldn't have to as many of the features involving password and secret management are available right here. Very easy.

Bitwarden's blue & white shield will now appear for fields that will accept log-in credentials.

Migrating passwords from Google Chrome and iCloud keychain

If you are coming from another password manager, you will likely have hundreds of passwords that need to go into your vault. Fortunately migrating to Vaultwarden is also pretty easy.

Google Chrome let's you export passwords saved in the password manager to a .csv file.

Once you have the file downloaded, in Vaultwarden, go to Tools > Import Data from the menu on the left. Select the correct file format (Chrome csv) and then upload the file.

I also needed to upload passwords from my iCloud keychain. You can only export passwords from the keychain in macOS. This will also output your passwords in a single .csv file and you can upload this to Vaultwarden.

Avoid using conflicting password managers

Don't forget to disable your previous password manager in your browser's settings. It seems like a “no-brainer,” but it is an important step. Otherwise, you will get some funky user interface presentation, especially in Chrome, and while trivial, this was a little annoying to deal with at first.

I do wish that Bitwarden had an option to disable the Chrome password manager entirely. I use multiple Google accounts and on each one I had to turn off the Chrome password manager. If that does exist, I wasn’t able to find it.

Passkeys

I was pleasantly surprised to find that Bitwarden manages passkeys! This allows for much faster log ins for platforms that accept them like you Google account.

Bitwarden on iPhone

To begin using Bitwarden on iOS, go to Settings > Passwords > password options and enable Bitwarden. Disable any other password managers you’re not planning on using.

Conclusion

Overall, this has been a smooth experience and I would recommend setting up and Vaultwarden if you’re looking for a secure & convenient way to store your passwords and secrets!

GitHub: https://github.com/dani-garcia/vaultwarden

So, why Proxmox?

Up until now, I've been using a Windows server with Hyper-V to run all my virtual machines. It worked fine, but lately, Proxmox has been catching my eye for a few reasons.

First of all, it can be used totally for free and it is an open-source project. That's a major plus for me. Along with that it has an engaged and active community as far as I can tell after reviewing YouTube videos and the forums. On top of that, Proxmox lets you cluster your servers and even migrate VMs while they're running. This means way less downtime and better fault tolerance, which is exactly what I'm looking for.

Networking bells and whistles can wait. Right now, my network setup is pretty basic. But down the line, I'm will explore Proxmox's support for VLANs and bonding (link aggregation).

Back up the VM

Let's start from the beginning. The first step is to back up the VM.

If your VM has snapshots, you need to merge the desired snapshot file (.avhdx) with the main .vhdx file. Alternatively, you can delete the snapshots in Hyper-V, which will merge them together. Once that's done, right-click on the VM in Hyper-V and choose "export".

I exported my VM to my Synology NAS. This VM is a headless Debian 12 instance that hosts a Docker environment.

Transfer the .vhdx file to Proxmox

Now we need to transfer the .vhdx file to the local Proxmox volume. I used FileStation in DSM to transfer the file using SFTP, but you can use any FTP client to do this.

Remote connections in Synology DSM.

After the connection is made, in FileStation, select the .vhdx file and copy it to the the /var/lib/vz/images directory on the server.

Create a new VM in Proxmox

It's time to create a new virtual machine in Proxmox.

You do not need to select the installation media.

After the VM is created, select it and go to "Hardware", "Hard Disk", and then click on "Detach". Select the "Unused Disk 0" and click "Remove".

Convert the .vhdx to a .img file

Now we will need to convert the .vhdx file we imported earlier into a usable format for Proxmox. We can do this from the Proxmox shell. The utility we will use to do is this is called qemu-utils.

Run apt install qemu-utils. If you see this warning, it may seem like the utility failed to install.

To verify, run which qemu-img. If which outputs the location the executable, then the utility has installed successfully.

Now run a command to convert the .vhdx to the .img. The format for this command is qemu-img convert -f vhdx -O raw /path/to/vhdx/file/ /path/to/img/file.

In my case it looked like this -

qemu-img convert -f vhdx -O raw /var/lib/vz/images/Ontario.vhdx /var/lib/vz/images/Ontario.img

Now we need to import the disk into the Proxmox storage directory. Replace the placeholder VM ID in the command below with the actual ID of your virtual machine. You can find the VM ID by navigating to the Proxmox web interface, selecting the desired VM, and looking at the numerical value on the left side of the VM's name.

qm importdisk 102 /var/lib/vz/images/Ontario.img local-lvm

Go back to Proxmox, select the "Unused Disk 0" and click "Edit", then click "Add".

Run and test!

After adding the hard disk we can run the VM and check to make sure it is working properly

You may need to open the boot menu with the "ESC" key and select the correct boot device.

You can also do this from the Proxmox interface.

The end.

Google's Chrome browser is great, but I have some issues with it. The feature I used the most was the password manager. Combined with the iCloud keychain on my iOS and macOS devices, I had a haphazard system for saving my credentials. However, using multiple Google accounts for different activities and devices quickly became a disorganized mess.

Google's password manager is per account, and I often found myself searching for credentials in another account. This was especially annoying on iOS, where switching accounts required logging out and back in. Sometimes, none of my saved passwords worked because, in my hurry from the last login session, I failed to update them properly.

I kept telling myself I'd fix this one day, but it took some time before I finally found the right solution.

Vaultwarden as a self-hosted option

I wanted to self-host my password manager for security reasons. Sure, I had already let Google and Apple handle my sensitive data because it was easy and built-in (and hard to avoid these days quite honestly). But, I wasn't ready to trust another company like Nord or LastPass to do that for me, and I definitely wasn't interested in paying for this kind of service.

After some research, I found the Vaultwarden project on GitHub.

I was very pleased to see clear documentation and an active community of users. I found out that my Vaultwarden server would be hosted in a docker container. That server would communicate with Bitwarden clients via the implementation of its API so that I can access my passwords from any browser or mobile device. I would need to set up a Bitwarden account to use the clients (but Bitwarden will not see my sensitive data).

That sounded good to me. After reviewing the installation instructions, I went ahead and attempted to deploy this solution for myself.

Preparation for Installation

I planned on using one of my Linux Debian virtual instances for this project.

I didn't have Docker installed on that instance so I had to take care of that first.

Once I ensured that the Docker engine was working ...

... it was time to set up the Vaultwarden container.

Vaultwarden installation

You can install Vaultwarden by running the docker command as specified in the Github. This will pull the container image and mount the volume for the container. The --restart unless-stopped option is a Docker restart policy that ensures a container is automatically restarted unless it is explicitly stopped by the user.

Vaultwarden will map to port 80 on the host machine.

docker pull vaultwarden/server:latest
docker run -d --name vaultwarden -v /vw-data/:/data/ --restart unless-stopped -p 80:80 vaultwarden/server:latest

Running sudo docker ps showed that Vaultwarden was up and running.

Sure enough, when I went to port 80 on my host I saw the login screen.

So far, so good.

Configuring an HTTPS proxy

I clicked on the option to create my Bitwarden account at the bottom. After entering my user info, I got an error.

"This browser requires HTTPS to use the web vault"

The wiki explains the error:

"For proper operation of vaultwarden, enablingHTTPSis pretty much required nowadays, since the Bitwarden web vault usesweb crypto APIsthat most browsers only make available in HTTPS contexts."

This explanation is alluding to the solution which is an HTTPS-to-HTTP reverse proxy. In fact, the wiki mentions services like CloudFlare for proxying the traffic. I went about this a bit differently.

I have set up a hub-and-spoke VPN with Tailscale that connects many of my devices in my homelab securely on my own mesh VPN. This was working great for remote access without exposing these devices to the public web. However, for this project, I would need to expose the container to web.

Tailscale Funnel works beautifully for this. The funnel allows you to route traffic from the wider internet to a local service running on a machine and there is a reverse proxy mode.

There are some important things you need to take care of first and if you want to use Funnel in Tailscale. I am not going to be covering Tailscale in this post. You can reference their excellent documentation (linked above) if you want to learn more.

I had already enabled Funnel on my node so I didn't need to do those steps again.

sudo tailscale funnel 3000 creates the funnel and you will see this output with the domain name if it the command is successful:

However, I later found out that the best way to do this is run the command with the

-bg option to ensure that the process will run in the background. Otherwise, sending SIGTERM (Ctrl + C) to recover your prompt in the shell will immediately close your funnel.

So I adjusted the command to sudo tailscale funnel -bg 3000

sudo tailscale funnel -bg localhost:80 enables the reverse proxy to the container.

With that done, I could successfully reach the container via HTTPS. I went on to create my Bitwarden account and view my empty vault.

Success!

This is part 1 of the tutorial. In part 2, I will continue where I have left off with configuring and setting up the vault.

Coming soon...