← Back to Research

Edge-to-Cloud Data Pipeline for a BLE Biosensor Platform

Edge-to-Cloud Data Pipeline for a Multi-Modal BLE Biosensor Platform

The lab already had a BLE client for multi-modal physiological recording. I built the half it was missing: getting the data to the server safely while recording is still in progress, and getting it to the right people afterwards.

Role
Student Researcher
Period
May 2026 – Present
Institution
North Carolina State University　Supervised by Dr. Abraham Vazquez

Why this matters

The old workflow was: finish recording, find the CSV on your own laptop, copy it to the server somehow. The problem was not the inconvenience — it was that every step could lose data quietly. People forgot. Laptops ran out of battery. Wi-Fi dropped mid-session, so a file was half-sent or re-sent whole. And once data reached the server it belonged to nobody: who recorded it, who may read it, who deleted it were all unanswerable. This is human-subject data; re-recording is not a matter of re-running a script.

What I did

  • Built a resumable incremental uploader: it sends data while recording is still going, and only the newline-terminated bytes the server has not yet acknowledged. Each file carries its own acknowledged byte offset, so a reconnect resumes from that byte instead of resending the file.
  • A partial final row is never sent. It sounds like a detail, but without it the server receives half a record mid-session — and downstream analysis cannot tell a genuinely short row from an unfinished one.
  • A retry sends the identical byte range and SHA-256. The server rejects gaps and treats an identical retry as a duplicate, which makes retrying safe by construction rather than by luck.
  • Uploads are woken by native filesystem events (FSEvents on macOS, inotify on Linux/Jetson) rather than polling — polling is either slow or power-hungry, and an acquisition rig can afford neither.
  • Completion is explicit: End Session and Quit writes a SHA-256 sidecar. Idle time is never treated as proof that acquisition has ended — to a filesystem, a subject taking a break and a finished experiment look identical.
  • Implemented accounts on the server with FastAPI and SQLAlchemy: data belongs to a person, and an administrator can retrieve it by user, by session, or by individual file. Deletion moves items to a trash with a retention countdown shown in the interface, rather than destroying them outright.
  • Added an activity log covering sign-ins, sign-outs, permission changes and upload batches, so an administrator can see who is online and who has recently uploaded.
  • Wrote a screenshot-by-screenshot manual for non-programmers. Not everyone who records data in the lab writes code — a system nobody can operate is no system at all.

The lab's existing BLE client only wrote data to the local disk. This work added everything downstream of that: getting the data to the server reliably while recording is still in progress, and giving it an owner once it arrives.

The whole path. Sensor data lands on the recording computer's disk first — recording never depends on the network, so a dropped connection costs nothing. The uploader sends the file in pieces while it is still growing and the server acknowledges each piece; the downloader then retrieves data by user and session.
The whole path. Sensor data lands on the recording computer's disk first — recording never depends on the network, so a dropped connection costs nothing. The uploader sends the file in pieces while it is still growing and the server acknowledges each piece; the downloader then retrieves data by user and session.
The uploader. Local write rate, upload rate and backlog are shown live, so it is immediately obvious whether data is keeping up. Below is per-file progress; on first run it asks whether to backfill pre-existing files, and that decision is remembered so they can be uploaded later on demand.
The uploader. Local write rate, upload rate and backlog are shown live, so it is immediately obvious whether data is keeping up. Below is per-file progress; on first run it asks whether to backfill pre-existing files, and that decision is remembered so they can be uploaded later on demand.
The administrator view — one row per account, expandable to a session or a single file. The slider filters by upload time. Deleting moves items to that account's own trash rather than destroying them.
The administrator view — one row per account, expandable to a session or a single file. The slider filters by upload time. Deleting moves items to that account's own trash rather than destroying them.
Account management and the activity log. Above, each account's presence, last upload and active login count; below, sign-ins, sign-outs and permission changes — so who owns data and who touched it stays answerable after the fact.
Account management and the activity log. Above, each account's presence, last upload and active login count; below, sign-ins, sign-outs and permission changes — so who owns data and who touched it stays answerable after the fact.

Technologies

  • Python
  • PyQt6
  • FastAPI
  • SQLAlchemy
  • BLE / bleak
  • Resumable upload
  • SHA-256 verification
  • FSEvents / inotify
  • Hydra