How to Build a Telerehabilitation Platform With Video and AI

Physical therapy has a dropout problem. Roughly 70% of patients fail to complete their prescribed rehabilitation programs, according to data from the American Physical Therapy Association. The reasons are predictable: long commutes to clinic locations, rigid scheduling that conflicts with work, high copays for in-person visits, and a lack of accountability between sessions. The result is poor outcomes, extended recovery timelines, and billions in avoidable downstream healthcare costs.

Telerehabilitation solves most of these friction points. Patients complete guided exercise sessions from home, therapists monitor form and adherence through video and sensor data, and AI-powered movement analysis catches compensation patterns that even experienced clinicians miss during a 30-minute in-person appointment. The clinical evidence is strong: a 2026 meta-analysis in the Journal of Orthopaedic and Sports Physical Therapy found that supervised telerehabilitation produced equivalent functional outcomes to in-clinic care for total knee replacement, rotator cuff repair, and chronic low back pain.

The market reflects this shift. Sword Health raised over $300 million and reached a $3 billion valuation by combining motion sensors with human clinical oversight. Hinge Health built a $6 billion business around digital MSK therapy with wearable sensors and a coaching model. Kaia Health uses smartphone cameras and computer vision to analyze patient movement in real time. These companies proved the model works. Now the opportunity exists for platforms that go deeper on specific patient populations, integrate more tightly with health systems, and deliver the therapist experience that these first-generation platforms often neglected.

If you are building a telerehabilitation platform in 2027, this guide covers the full technical and clinical stack: HIPAA-compliant video infrastructure, exercise prescription engines, AI movement analysis, wearable integration, therapist dashboards, billing workflows, and the regulatory landscape you need to navigate before writing a single line of code.

Video is the backbone of synchronous telerehabilitation. Unlike a standard telemedicine visit where a patient sits in front of a camera and talks, rehab sessions require the patient to stand, move through exercises, and position their body in ways that the therapist can observe from multiple angles. This creates unique technical requirements that generic video conferencing cannot handle well.

Your video infrastructure must support low-latency, high-definition streaming so therapists can observe joint angles, detect compensatory movements, and provide real-time verbal and visual cues. A 200ms delay that is barely noticeable in a conversation becomes a problem when a therapist is cueing a patient through a squat pattern and the feedback arrives after the rep is complete.

Choosing Your Video Infrastructure

Daily.co is the strongest option for healthcare startups building telerehabilitation today. Their API is clean, their HIPAA compliance posture is mature (they sign BAAs and offer encrypted cloud recording to customer-managed S3 buckets), and their pricing is transparent at roughly $0.004 per participant-minute. For a typical 45-minute rehab session with two participants, that is about $0.36 per session. At 10,000 sessions per month, you are looking at $3,600 in video infrastructure costs.

Twilio Video remains a solid alternative, especially if you are already using Twilio for SMS reminders and voice calls. Their per-participant-minute pricing runs slightly higher, but the unified billing and SDK consistency across communication channels simplifies your vendor stack. Both Twilio and Daily.co support WebRTC with adaptive bitrate streaming, which is critical for patients on home Wi-Fi or cellular connections.

For telerehabilitation specifically, your video implementation needs capabilities beyond what a standard telemedicine app requires:

• Wide-angle camera support: Patients need to position themselves far enough from their device for the therapist to see full-body movement. Your UI should guide camera placement with an overlay showing the ideal capture zone, similar to what Kaia Health does during onboarding.
• Dual video streams: Support simultaneous display of the patient's live feed and a pre-recorded exercise demonstration, so therapists can show proper form side-by-side with the patient's movement.
• Session recording with consent: Record sessions to encrypted storage for clinical documentation, quality assurance, and AI model training (with explicit patient consent). Recordings must be AES-256 encrypted at rest with access logging per HIPAA requirements.
• Bandwidth resilience: Graceful degradation to lower resolution before dropping to audio-only. Many rehab patients are older adults on inconsistent home internet connections. Your platform cannot fail silently when bandwidth drops.
• Annotation tools: Let therapists draw on the video feed to highlight body positioning, point to specific joints, or mark range-of-motion targets. This visual feedback is far more effective than verbal cues alone.

Build your video layer as a modular service that your synchronous sessions and your asynchronous exercise review workflows both consume. Therapists should be able to review recorded patient exercise sessions and leave timestamped video comments without scheduling a live call. This hybrid sync/async model is how the best telerehab platforms maximize therapist capacity while maintaining clinical quality.

Every telerehabilitation platform needs a structured exercise library. This is not a YouTube playlist. It is a clinically organized, searchable database of therapeutic exercises with standardized metadata that powers your prescription engine, your AI analysis pipeline, and your patient-facing exercise player.

Your exercise library should include the following data for each exercise:

• Clinical metadata: Target body region (shoulder, knee, lumbar spine), movement type (strengthening, stretching, balance, proprioception), difficulty level (1 through 5), contraindications, and applicable diagnosis codes (ICD-10).
• Video demonstrations: Professional-quality recordings showing proper form from at least two angles (front and side). Include common error demonstrations so your AI model can learn what incorrect form looks like. Each video should be 15 to 45 seconds with clear verbal cues overlaid.
• Pose reference data: Skeleton keypoint data for the correct movement pattern, captured using MediaPipe or a similar pose estimation framework. This reference data becomes the ground truth for your AI movement analysis engine.
• Prescription parameters: Default sets, reps, hold times, rest intervals, and progression criteria (for example, "advance to Level 3 when patient completes 3 sets of 12 with less than 2/10 pain").

Start with 200 to 400 exercises covering the most common rehabilitation pathways: post-operative knee and hip (total joint replacement), shoulder (rotator cuff repair, frozen shoulder), lumbar spine (disc herniation, stenosis), and general deconditioning. Sword Health launched with roughly 500 exercises. Hinge Health has expanded to over 1,000. You do not need that volume at launch, but your data model must support it.

Building the exercise prescription workflow: Therapists should be able to build custom programs by searching and filtering the exercise library, dragging exercises into a weekly schedule, setting parameters for each exercise, and adding text or video instructions specific to the patient. The prescription engine should support templates: a "standard post-TKR week 1-2 protocol" template that a therapist can assign with minor modifications saves 15 minutes per patient compared to building every program from scratch.

On the patient side, the exercise player should display one exercise at a time with the demonstration video, written instructions, a timer or rep counter, and a pain/difficulty rating after each exercise. This session data feeds back into your analytics and drives the therapist's clinical decision-making. Patients who report increasing pain during a specific exercise should trigger an automated alert to the therapist, not wait until the next scheduled visit for review.

License your initial video content from clinical content libraries or partner with a physical therapy program at a university. Filming 300+ exercises with professional videography, licensed PTs demonstrating proper form, and post-production editing will cost $40,000 to $80,000. This is a significant upfront investment, but the exercise library is a core competitive asset and a moat against competitors who rely on generic content.

AI movement analysis is the feature that separates a telerehabilitation platform from a video call with homework. By using computer vision to track a patient's body position during exercises, you can provide real-time feedback on form, measure range of motion objectively, detect compensation patterns, and track progress over weeks and months with quantitative data that supplements the therapist's clinical judgment.

Pose Estimation Pipeline

Google MediaPipe is the most practical foundation for client-side pose estimation in 2027. MediaPipe Pose detects 33 body landmarks in real time and runs efficiently on mobile devices without requiring a GPU. The BlazePose model underneath achieves roughly 30 FPS on a mid-range smartphone, which is sufficient for tracking rehabilitation exercises that are performed at controlled speeds. For web-based sessions, MediaPipe's JavaScript SDK runs pose estimation directly in the browser via WebAssembly and WebGL, eliminating the need to stream video to a server for analysis.

Your movement analysis pipeline works like this:

• Frame capture: Extract video frames from the patient's camera feed at 15 to 30 FPS during exercise execution.
• Pose detection: Run MediaPipe Pose on each frame to extract 33 skeletal keypoints with x, y, z coordinates and confidence scores.
• Joint angle calculation: Compute angles between relevant joints using the keypoint coordinates. For a knee flexion exercise, calculate the angle between the hip, knee, and ankle landmarks. For shoulder abduction, calculate the angle between the torso midline and the upper arm vector.
• Movement comparison: Compare the patient's joint angles and movement trajectory against the reference data from your exercise library. Flag deviations that exceed clinically meaningful thresholds (typically 10 to 15 degrees for most joints).
• Feedback generation: Provide real-time visual and audio cues when form deviates. "Straighten your back" when lumbar flexion exceeds the threshold during a squat. "Lift your arm higher" when shoulder elevation falls short of the target range.

Beyond Basic Pose Estimation

Raw pose estimation is the starting point, not the finish line. The real clinical value comes from building domain-specific models on top of the pose data. Train classification models to detect specific compensation patterns: a patient favoring their non-surgical leg during a step-up, excessive trunk lean during single-leg balance, or scapular winging during overhead reaching. These models require labeled clinical data, which means partnering with physical therapists to annotate thousands of exercise repetitions as correct, compensated, or incorrect.

For range-of-motion (ROM) measurement, your system needs to be accurate within 5 degrees to be clinically useful. Academic validation studies have shown that MediaPipe-based ROM measurement achieves 3 to 7 degree accuracy compared to goniometer measurements, depending on the joint and the viewing angle. That is acceptable for progress tracking but not yet sufficient for surgical decision-making. Be transparent about these limitations in your clinical documentation.

Processing architecture: Run pose estimation on-device for real-time feedback during exercise sessions. Store the raw keypoint time-series data (not the video) server-side for longitudinal analysis, therapist review, and model improvement. A 30-minute exercise session generates roughly 2 to 5 MB of keypoint data, compared to 500 MB or more for the raw video. This makes server-side storage and analysis economically viable at scale.

Be aware that AI movement analysis in a healthcare context may trigger FDA scrutiny. We cover the regulatory implications in the section on FDA and SaMD classification below.

Quantitative outcome tracking is what makes telerehabilitation defensible against the perception that it is "less than" in-person care. Your platform should generate objective progress data that therapists can use for clinical decisions and that payers can use to justify reimbursement.

Outcome Measures and Progress Tracking

Build standardized patient-reported outcome measures (PROMs) directly into your platform. The most commonly used instruments in rehabilitation include the Visual Analog Scale (VAS) for pain, the Oswestry Disability Index (ODI) for low back pain, the DASH questionnaire for upper extremity function, and the KOOS for knee outcomes. Automate PROM collection at clinically appropriate intervals (weekly for acute post-surgical patients, biweekly for chronic conditions) and display trend lines on the therapist dashboard alongside objective movement data.

Combine PROMs with the quantitative data from your AI movement analysis: range-of-motion trends, exercise completion rates, session frequency, pain ratings per exercise, and rep quality scores. This multi-dimensional view of patient progress is more comprehensive than anything a therapist gets from in-person visits, where they see the patient for 30 minutes and rely on self-reported home exercise compliance for the other 167.5 hours of the week.

Wearable Device Integration

Wearable devices add a continuous data stream between therapy sessions. Apple Watch and Fitbit are the most common consumer wearables your patients will already own. Both expose health data through their respective APIs (Apple HealthKit on iOS, Fitbit Web API via OAuth 2.0) that is directly relevant to rehabilitation:

• Step count and activity minutes: Track overall activity levels and detect sudden declines that may indicate a setback or flare-up.
• Heart rate data: Monitor cardiovascular response during exercise sessions to ensure patients stay within safe zones, particularly important for cardiac rehabilitation.
• Range-of-motion data: Apple Watch Series 9 and later support wrist ROM measurement, and third-party apps on both platforms can estimate shoulder and elbow ROM using accelerometer and gyroscope data.
• Sleep data: Sleep quality correlates strongly with pain levels and recovery speed. Therapists can use sleep trend data to contextualize patient-reported outcomes.

Sword Health ships its own motion sensor that straps to the body part being rehabilitated, giving them clinical-grade movement data. If you want similar precision without manufacturing hardware, consider integrating with APDM Wearable Technologies (now part of Clario) or BioSensics, which make FDA-cleared inertial sensors used in clinical research. These sensors provide joint angle accuracy within 1 to 2 degrees, far exceeding what consumer wearables or camera-based pose estimation can achieve. For a deeper look at wearable data architecture and HealthKit integration patterns, see our guide on building a wearable health app.

Therapist Dashboard and Scheduling

The therapist dashboard is where clinical efficiency is won or lost. Design it around the concept of a patient panel, similar to how primary care EHRs organize patient lists. Each therapist should see their active patients sorted by priority: patients with declining outcomes at the top, patients due for program progression in the middle, and stable patients at the bottom.

Key dashboard views include:

• Patient summary cards: At-a-glance view showing last session date, exercise adherence percentage (completed sessions / prescribed sessions), current pain level, and ROM trend (improving, stable, declining).
• Asynchronous review queue: Recorded exercise sessions flagged for therapist review, prioritized by the AI's confidence score. Low-confidence movement assessments get reviewed first.
• Scheduling panel: Calendar view with availability management, appointment types (initial evaluation, follow-up, group session), and automated patient reminders via SMS and push notification.
• Clinical documentation: SOAP note templates pre-populated with session data, ROM measurements, and PROM scores. A therapist should be able to complete documentation in under 3 minutes per visit.

Build the dashboard as a responsive web application using Next.js or a similar framework. Therapists work from laptops and desktops, not phones. Prioritize keyboard navigation, fast search, and dense information display over visual flourish. For scheduling specifics, our guide on building a telemedicine app covers the core scheduling engine architecture in detail.

Reimbursement is what makes a telerehabilitation platform financially viable for the clinics and health systems that adopt it. If therapists cannot bill for the services they deliver through your platform, adoption stalls regardless of how good the technology is.

CPT Codes for Telerehabilitation

The most commonly billed CPT codes for synchronous telerehabilitation sessions include:

• 97110 (Therapeutic exercises): The workhorse code for rehab sessions focused on strength, endurance, flexibility, and range of motion. Billed in 15-minute units.
• 97530 (Therapeutic activities): Functional movement training with direct patient contact. Also billed in 15-minute units.
• 97542 (Wheelchair management training): Applicable for patients using assistive devices during telerehab sessions.
• 98966-98968 (Telephone E/M): For asynchronous consultations and phone-based follow-ups, billed by duration.
• Modifier 95 or GT: Appended to indicate the service was delivered via synchronous telehealth. Payer requirements vary; some require modifier 95, others require GT, and some accept either.

Your billing module should map each session type to the appropriate CPT codes, calculate units based on session duration (the 8-minute rule applies: you need at least 8 minutes of direct patient contact to bill one unit), and generate claims with the correct modifiers and place-of-service codes. Place of service 02 (Telehealth) or 10 (Telehealth in patient's home) are used depending on the payer's requirements.

Integrate with a clearinghouse like Availity, Change Healthcare, or Waystar for electronic claims submission, eligibility verification, and remittance processing. These integrations cost $500 to $2,000 per month and save your customers from manual claims entry, which is error-prone and slow.

FDA and SaMD Classification

This is where many telerehabilitation startups underestimate the regulatory landscape. If your AI movement analysis provides clinical recommendations, such as suggesting exercise modifications based on detected compensation patterns or flagging patients for therapist review based on ROM decline, it may qualify as a Software as a Medical Device (SaMD) under FDA guidance.

The FDA's 2023 framework for AI/ML-based SaMD uses a risk matrix based on the seriousness of the condition and the significance of the information the software provides. A telerehabilitation platform that provides movement feedback to patients during exercises is likely Class I or Class II, depending on the claims you make. If you claim your AI "diagnoses" movement disorders or "prescribes" treatment modifications, you are firmly in Class II territory and need 510(k) clearance. If your AI provides "wellness" feedback and all clinical decisions require therapist review, you may qualify for an exemption under the FDA's enforcement discretion policy for clinical decision support software.

Work with a regulatory consultant who specializes in digital health SaMD before you finalize your product claims. Firms like Emergo by UL or Greenlight Guru can help you navigate the classification process and develop a regulatory strategy. Budget $50,000 to $150,000 for FDA regulatory preparation if a 510(k) submission is required, plus 6 to 12 months for the review timeline.

Hinge Health and Kaia Health both navigated this regulatory landscape successfully. Hinge Health's Enso device received FDA clearance as a Class II medical device. Kaia Health has published peer-reviewed clinical validation studies that support their approach. Study their regulatory filings and clinical evidence strategy as you build your own.

State Licensing and Telehealth Parity

Physical therapists must be licensed in the state where the patient is located during the session. If your platform serves patients across multiple states, your therapists need licenses in each state, or you need to operate under the Physical Therapy Licensure Compact, which currently covers 40+ states. Your platform should verify that each therapist-patient pairing is compliant with state licensing requirements before allowing a session to be scheduled. Build this as a rules engine that flags non-compliant pairings during scheduling rather than discovering the issue after the session has been delivered and billed. Our guide on building a healthcare app covers HIPAA infrastructure and compliance architecture in more detail.

The best technology in the world fails if patients do not use it consistently. Telerehabilitation adherence hinges on engagement tools that keep patients motivated between live sessions and a rollout strategy that builds clinical confidence before scaling.

Patient Engagement Tools

Build automated reminders that go beyond simple appointment notifications. Your reminder system should include:

• Exercise session reminders: Push notifications at the patient's preferred time with a one-tap launch into their daily exercise program. SMS fallback for patients who disable push notifications.
• Streak tracking and milestones: Display consecutive days of exercise completion, celebrate milestones (7-day streak, 50% program completion), and show progress toward discharge goals. Gamification works in rehabilitation when it is tied to meaningful clinical outcomes, not arbitrary points.
• Therapist check-ins: Automated prompts for therapists to send brief video or text messages to patients who have missed 2+ consecutive sessions. Personal outreach from a therapist is 3x more effective at re-engaging patients than automated reminders alone, based on data from Hinge Health's published engagement studies.
• Educational content: Short articles and videos about the patient's condition, what to expect during recovery, and self-management strategies. Deliver these at clinically appropriate points in the recovery timeline. A patient two weeks post-ACL reconstruction should see content about managing swelling, not content about return-to-sport timelines.
• Pain and symptom journaling: Daily 30-second check-ins where patients log their pain level, sleep quality, and any concerning symptoms. This data feeds the therapist dashboard and can trigger automated alerts when patterns indicate a potential complication.

Phased Rollout Strategy

Do not try to launch everything at once. A phased approach reduces risk and lets you validate assumptions before scaling:

Phase 1 (Months 1 through 4): Foundation. Build the core video session infrastructure, exercise prescription engine with 200+ exercises, basic patient progress tracking, therapist dashboard, and HIPAA compliance framework. Cost: $150,000 to $250,000. Launch with 3 to 5 pilot clinics in a single specialty (post-surgical knee rehab is ideal because outcomes are measurable and the patient population is motivated).

Phase 2 (Months 5 through 8): Intelligence. Add AI movement analysis with MediaPipe pose estimation, wearable device integration (Apple HealthKit and Fitbit), automated PROM collection, and asynchronous exercise review for therapists. Cost: $100,000 to $180,000. Expand to 15 to 20 clinics and add a second clinical pathway (shoulder rehabilitation or chronic low back pain).

Phase 3 (Months 9 through 12): Scale. Integrate insurance billing with a clearinghouse, add group therapy sessions (one therapist supervising 4 to 6 patients simultaneously), build the clinical outcomes reporting module for value-based contracts, and pursue FDA regulatory clearance if your product claims require it. Cost: $120,000 to $200,000. Target health system partnerships and payer contracts.

Total investment for a full-featured telerehabilitation platform: $370,000 to $630,000 over 12 months. Ongoing operational costs (cloud infrastructure, video API, wearable API subscriptions, compliance maintenance) run $8,000 to $18,000 per month.

The telerehabilitation market is projected to exceed $15 billion by 2030, driven by an aging population, rising MSK care costs, and payer willingness to reimburse virtual PT at parity with in-person care. The platforms that win will combine clinical rigor with consumer-grade user experience, quantitative outcome tracking, and tight integration with the existing healthcare ecosystem. If you want to build for this market, start with a clear clinical use case, validate with real therapists and patients, and invest in the compliance and AI infrastructure that will differentiate your platform from a video call with a PDF exercise handout.

We have built HIPAA-compliant healthcare platforms with real-time video, AI analytics, and clinical integrations. If you are ready to move from idea to architecture, book a free strategy call and we will help you scope your MVP, select vendors, and map out a realistic timeline to market.